Fatigue driving detection based on electrooculography: a review

Abstract

To accurately identify fatigued driving, establishing a monitoring system is one of the important guarantees of improving traffic safety and reducing traffic accidents. Among many research methods, electrooculogram signal (EOG) has unique advantages. This paper presents a systematic literature review of these technologies and summarizes a basic framework of fatigue driving monitoring system based on EOGs. Then we summarize the advantages and disadvantages of existing technologies. In addition, 80 primary references published during the last decade were identified. The multi-feature fusion technique based on EOGs performs better than other traditional methods due to its low cost, low power consumption and low intrusion, while its application is still limited which needs more efforts to obtain good and generalizable results. And then, an overview of the literature on technology is given, revealing a premier and unbiased survey of the existing empirical research of classification techniques that have been applied to fatigue driving analysis. Finally, this paper adds value to the current literature by investigating the application of EOG signals in fatigued driving and the design of related systems, future guidelines have been provided to practitioners and researchers to grasp the major contributions and challenges in the state-of-the-art research.

Introduction

According to the global status report on road safety conducted by the World Health Organization (WHO) in 2015, 1.25 million people die from traffic accident every year worldwide and millions of more people suffer serious injuries in traffic accidents. It is estimated that the main cause of death of young people is road traffic injures, especially those who aged 15–29 years [1]. During the driving process, the driver’s mental state seriously affects the driving behavior, which is a major hidden danger of traffic safety driving [2]. In Europe, a study of 19 countries showed that 17 percent of drivers felt drowsy when driving, while 7% of those who fell asleep were involved in an accident [3].

In other kinds of illegal driving situations, such as overloading, speeding, overcrowding, drunk driving, we can take effective measures to supervise and reduce the occurrence of accidents. However, there are no quantitative and scientific measurement standards of fatigue driving. Different from other illegal driving behaviors, fatigued driving reflects a complex operation behavior, and it is difficult to be judged and identified. How to correctly and scientifically define driving fatigue and implement more accurate monitoring in real time is the main research focus for researchers.

The European Transport Safety Council (ETSC) defines the fatigue as “concerns the inability or disinclination to continue an activity, generally because the activity has been going on for too long” [4]. There are four different kinds of fatigue according to the ETSC: local physical fatigue, general physical fatigue, central nervous fatigue and mental fatigue. Sleep in humans is considered as a passive process, whereas sleep actually is an active process [5]. Sleep in humans is generally divided into two types: rapid eye moment (REM) sleep and non-rapid eye movement (NREM) sleep [5]. Although researchers have made significant progress for indicating fatigue status, there are still very few findings that can be put into practice because of the environmental complexity and under-developed mechanisms of fatigue in the transport field [6].

Operator fatigue analysis contains diverse research questions. A reliable fatigue detection device in the future is not only about the development of a fatigue detection algorithm and hardware in mechanical engineering, but also about the understanding of the underlying mechanism of fatigue in a transport scenario. There are many existing fatigue monitoring technologies and classification methods, which can be divided into five categories according to the detection parameters and application technologies: (1) subjective measures: these methods are applied to measure the fatigue degree of the driver through drivers' subjective self-assessment; (2) driver’s biological measures: these methods are applied to detect fatigue by detecting changes of drivers’ biological signals, including electrocardiogram (ECG) [7], electroencephalogram (EEG) [8, 9], EOG [10] and other fatigue detection methods; (3) driver physical measures: these methods are applied to detect fatigue degree by human physiological response [11,12,13,14]; (4) driving performance measures: these methods are applied to judge the fatigue degree of drivers by driving performance [15]; (5) hybrid measures: these methods are often applied to integrate the two or more detection methods mentioned above to improve the monitoring accuracy [16,17,18] to deal with larger data and have better performance.

The existing methods of driving fatigue monitoring have some limitations. Although the subjective evaluation method is widely used, it has inherent defects which include the expectation bias and it can interfere with normal work, not suitable for continuous real-time monitoring of mental fatigue. Moreover, it is unrealistic to always ask drivers to report their state [19]. The vision-based monitoring method is greatly affected by lighting. In the case of sunlight reflection and glasses reflection, the performance may decrease by 30% [20]. In the current vision-based monitoring methods, the work for frontal faces is good, yet extreme head posture will lead to errors in monitoring results [21]. Monitoring driver behavior based on vehicle operation information requires modification of vehicle structure, which is unrealistic and unwise in reality [20]. The development of vehicle automation also has a certain restrictive effect on the technology of obtaining fatigue state by vehicle movement [6]. In the monitoring method based on physiological signals, EOG is a potential fatigue monitoring technology due to its relatively low cost, low power consumption, corresponding speed, and it does not block the driver’s vision. In addition, EOG also has extensive eye movement tracking capability.

There is no doubt that the previous reviews on drivers’ fatigue monitoring collectively covered a wide range of fatigue systems, but those reviews vary in scope [22, 23]. Comparatively, the main contributions of this review on this subject is as below:

• 1. The importance of mental fatigue and fatigue driving is discussed and emphasized.

• 2. The limitations of fatigue driving monitoring methods in the current transportation field are discussed and analyzed to simplify the current research on non-invasive fatigue monitoring.

• 3. This review adds value to the current literature by studying the application of EOG signals in fatigued driving and the design of related systems.

In the rest of this review, the research results of the latest fatigue driving monitoring technology based on EOG are discussed in the second section. Next, the feature extraction and correlation extraction methods of EOG signals are discussed in the third section. And then, the feature classification method and the fatigue driving system based on EOG are discussed in the fourth section. Finally, several future interests with a discussion regarding the methodology of conducting operator fatigue research are provided.

Related work

In the past few years, there have been a number of fatigue driving monitoring techniques based on a variety of features. In the traditional methods, various signals generated by the driver are regarded as the effective characteristics of fatigue monitoring. Different from other bioelectrical signal, as its great potential of practicality and economy, the system based on EOGs has been developing rapidly in the field of fatigue driving condition monitoring in recent years.

A theoretical framework of fatigue driving system based on EOGs using the current technology and equipment is summarized. The structure of the proposed framework is illustrated in Fig. 1. The recognition technology based on biological signal belongs to one of the objective detection methods which is mainly applied to judge whether the driver is in the state of driving fatigue by analyzing the change rule of biological signal of the driver's body. The physiological status can be detected by supervising driver’s related physiological parameters, such as eyes activity, facial expressions, head nodding, body sagging posture, physiological electrical signals, etc.

Fig. 1

Structure of the fatigue monitoring framework

As one of the physiological and electrical signals of the human body, EOG contains abundant information on features of eyelid movement. Different types of eye movement features can be extracted from EOGs, including REM [24], slow eye movement (SEM) [25], and eyelid movement [26], etc. To solve the problem that the electrode placement of the traditional EOG, Zhang et al. proposed a novel electrode placement on forehead to extract horizontal electrooculogram (HEO) and vertical electrooculogram (VEO) from forehead EOG [27]. In addition to being used alone to monitor mental fatigue, EOGs are often combined with other signals to form new quantitative tools for mental fatigue [28, 29]. Ahn et al. used multimodal EEG/ECG/EOG and functional near infra-red spectroscopy (fNIRS) data to explore drivers’ mental fatigue states [30]. Picot et al. used both brain and visual activity to detect drowsiness which enabled the false alarm rate to be reduced to 5% [31]. In the field of machine learning, Jiao et al. [32] proposed a method, some new features were extracted based on wavelet singularity analysis and statistics to detect SEMs. Classifiers such as SVM, the discriminative graph regularized Extreme Learning Machine (GELM), and KNN were compared with a 2 s HEO signal which could finally be recognized as the category of SEMs or non-SEMs. The combination of the wavelet energy features and the new features based on wavelet singularity analysis and statistics improved the detection performance. The man–machine response mode (MRM) based on EOG and other eye movement characteristics can relieve long-term driving fatigue [2].

With the many studies already done on fatigue driving monitoring based on EOGs and the importance of these findings and their contribution to traffic safety, these kinds of work are the most essential to be explored as they laid the foundation for future technology.

Extraction of EOG features

Driving fatigue detection is a nonlinear problem. Physiological signals such as EEG and EOG are not fixed, and EOG is considered as a potential and effective method of monitoring fatigue status. Different physiological signals have different characteristics, such as the baseline drift of EOG and the non-repeatability and high signal-to-noise ratio (SNR) [33], the repeatability and good adaptability to baseline drift of ECG. In recent years, feature extraction methods for different physiological signals have also made considerable headway.

Baseline drift of EOG

Implementing an EOG-based interface raises several problems. Baseline drift is a major concern, since it decreases the tracking accuracy. Although drift changes the baseline slowly and it can be ignored over short periods, it can’t be ignored over longer periods, such as fatigue driving monitoring which can give an inaccurate absolute eye angle.

There are several strategies to overcome the baseline drift. The first strategy tries to lower the impact of the drift using a wavelet transform [34] and multiple EOGs [35]. This strategy is effective in extending the period of ignoring the drift not for a long period. Accoupled EOGs are an easy solution to the drift problem which can be used to estimate the absolute eye angle [36]. However, once an estimation error occurs, the error will persist all the time. Different from the previous strategy, frequent offset calibration can offset the drift [37]. However, the current methods require explicit effort which can interrupt the main function. Finally, tuning the interaction is the other solution. The drift can be ignored when the interaction is based on the short-time eye movements instead of absolute eye angle when it needs an additional eye gesture.

As indicated previously, a single solution strategy is often unable to completely solve the drift problem. Thus, they are often combined in practical systems to deal with the drift issue. EOGs are useful for estimating eye gaze but it can add artifacts to EEG [38, 39]. Many researchers tried to remove EOG components from other signals [40, 41]. Zahan proposed a method based on the independent component analysis and multivariate empirical mode decomposition to remove EOG artifacts from EEGs [42]. Cheng et al. used a combination of singular spectrum analysis and second-order blind identification method to remove diverse artifacts [43]. However, EOGs and other bioelectrical signals bidirectionally contaminate each other [44]. Unlike EEG, the nonlinearity of EOG is small, but the drift problem will eventually reduce the accuracy of the system when researchers regard it as a linear problem. Manabe et al. proposed a gaze estimation technique based on the nonlinearity between the EOG and eye angle which provided a practical solution to the drift problem [45].

Preprocessing of EOGs based on wavelet transform

Existing techniques for processing signals include Fourier transformation (FFT) [46], short-time Fourier transformation (STFT) [47], discrete wavelet transformation (WT) [48], wavelet packet transformation (WPT) [49], and entropy [50]. Wavelet transform is more suitable than FFT for unstable signal processing, such as EEG and EOG, and Shannon entropy is more suitable for compression than classification problems. In fatigue driving EOG signal processing, there are a lot of researches based on wavelet transform.

Ho et al. used the wavelet methodology for EOG state classification which emphasized wavelet pattern recognition methods [51]. The EOGs were processed by WT and then produced fuzzy signals for a neural network. High precision in time localization in the high frequency band can be achieved at the expense of reduced frequency resolution in WT and it can be scaled to match most of the high and low frequency signals. Therefore, it can achieve the optimal resolution with the least number of base functions. Calculating the percentage of eye closure over time (PERCLOS) is not robust in practical applications due to the complexity of eye detection methods [26]. WT is sensitive to singularity and can produce a better result than the derivative method. While highly dynamical alterations are better reflected by EOGs than by integral measures such as PERCLOS for not containing any assessment of eye and eyelid movements. Few methods can detect overlong eye lid closures (more than 3 s) [52]. New features were extracted based on wavelet singularity analysis and statistics to detect SEMs which can improve the classification results. For traditional EOGs, WT can be used to obtain many features which can be classified with higher classification accuracy [53, 54].

Preprocessing of EOGs based on wavelet packet

Using the WPT can construct features correlate with alertness and drowsiness. Different from WT, the WPT not only decomposes the approximation coefficients and the detail coefficients, but also can deal with stationary, nonstationary, or transitory characteristics of different signals [55].

The WPT was introduced by Coifman et al. [56] by generalizing the link between multiresolution approximations and wavelets. WPT has more subsequent decomposition levels than WT. Zhang et al. [57] utilized the relative energies of the WPT subspaces with Shannon entropy as a measure for feature suitability for detecting drowsiness which called HWPT. Wang et al. [58] proposed the optimal wavelet-packet feature-extraction method which called OWP. This method produced a better result. Khushaba et al. proposed a fuzzy mutual-information (MI)-based WPT feature-extraction method for classifying the driver drowsiness state [59]. This method estimated the required MI using the fuzzy memberships which achieved a classification accuracy of 95–97% on an average across all subjects. Vigilance states are intrinsic mental states that involve temporal evolution rather than a time point [60]. EOGs are easier to implement and ultimately more feasible than EEG for large-scale implementations for its higher signal-to-noise ratio. Zheng et al. used WPT to extract features from the forehead EOGs and applied two temporal dependency models, continuous conditional neural field (CCNF) and continuous conditional random field (CCRF) [53]. Ding et al. [49] extracted different features including WPT coefficient and WT coefficient for EOG signals and the experimental results indicated that feature-level fusion (FLF) strategies achieved better classification accuracies than decision-level fusion (DLF) strategies.

Other methods of EOGs preprocessing

Different from the approach methods mentioned above, traditional linear spatial filters include principal component analysis (PCA), independent component analysis (ICA) [61], common spatial patterns for BCI applications [62] or beamforming methods for source localization [63]. Chambon et al. used the linear spatial filters to exploit the array of sensors which increased the SNR [64]. The last layer fed the features to a softmax classifier. As signal-gathering tools have improved, specialized manufacturers have produced devices that can quickly extract features. Kim et al. used the teager energy operator (TEO) by differential operator circuit with high-pass filter and low-pass configurations to extract features from multiple physiological signals [65]. Li et al. used a wearable eye tracker to acquire a number of eye-movement features without signal preprocessing process [66]. Advanced driving simulator can easily obtain a large number of signal features required by fatigue driving monitoring algorithm in laboratory environment [67].

With the many studies already done on extracting features from EOGs and the importance of these findings and their contribution to work, great progress has been made in fatigue driving feature extraction technology based on EOGs. Meanwhile, these methods also promote the research and development of fatigue driving system based on EOG signals.

Processing of EOGs and classifier models

Theoretical frameworks are essential contributions in grasping insight about the mechanisms influencing fatigue driving monitoring, and in the development of improving transportation safety [68]. A theoretical framework for fatigue driving detection systems is summarized to more specifically study the relevant issues in this field. Accordingly, practitioners will be better directed in the future development of intervention strategies designed to improve safety outcomes.

The driver, vehicle and environment model (DVE model) are based on the concept of the ‘‘joint’’ cognitive system, where the dynamic interactions of the DVE model are represented in a complex way. This model can be seen a as closed loop system whose impact on the driver is observed in a driving situation.

Framework design of systems based on EOGs

In the DVE system, there are lots of actions of drivers to control the vehicle based on current driving environment. Fatigue driving monitoring technology has been greatly developed in recent years based on various body information of the driver during the driving process. As a potential approach of bioelectrical signal, EOGs plays an important role in fatigue driving condition monitoring system. A certain number of basic components affecting variability in driver’s driving monitoring systems based on EOGs are identified and listed as follows:

Chen et al. [69] improved the traditional feature selection methods of EEG, ECG and EOG. Gao et al. [70] proposed a method of fatigue detection by eye tracking glasses to evaluate driving fatigue detection algorithms. This method provides a new option for wireless and convenient detection of fatigue monitoring. Damousis et al. [71] proposed a fuzzy expert system for the driving fatigue detection. A system could predict fatigue accurately with a fuzzy combination of eyelid activity parameters. Barua et al. proposed a method of considering the effect of individual differences, that accuracy increased 10% [72].

Considering the various parameters associated with mental fatigue abound, there are few revealed on how mental fatigue influences drivers’ ability to detect hazardous situations. Unlike electrophysiological signal monitoring, eye tracking can be unaffected by electromagnetic, temperature and vibration. The accuracy of physiological signals has attracted the attention of many researchers, but the visual examination of continuous physiological signals is still a difficult and challenged task even for trained neuroscientists.

Although all of these research papers provide a significant contribution to furthering our work of the fatigue monitoring, there is still a lack of broad overviews based on relevant theoretical frameworks [73].

Signal analysis and modality

Eye movement characteristics, such as saccades, fixations, and blinks detected by EOG signals, have already been used for fatigue driving monitoring [52]. The work detecting the three eye-movement characteristics demonstrates the promise of eye-based activity recognition (EAR) [33]. There are a large number of parameters obtained by the preprocessing of EOGs, researchers usually need to select appropriate characteristic parameters when using the model to monitor fatigued driving. Table 1 describes some of the features of eye movements commonly used in fatigue drive monitoring after preprocessing of electrical eye signals. Other eye movement characteristics such as pupil dilation, microsaccades, vestibulo-ocular reflex, or smooth pursuit movements have few applications for its hard-to-measure with EOG [33].

Table 1 Commonly eye movement features from EOGs

This section presents a categorization of EOG modality regarding the various processing models. Table 2 shows the performance and accuracy of the fatigue driving detection model based on EOG as a comparison method of performance.

Table 2 Performance and accuracy of the fatigue driving detection model

Feature classification

In fact, fatigue driving monitoring technology is the artificial construction of some algorithms, through the computer ability to automatically process the monitored data, extraction and classification, and automatic classification. Then the function of high precision fatigue monitoring can be realized [74,75,76,77]. Combining multiple non-invasive fatigue methods would provide good reliability [78]. Just like the frame design mentioned in Fig. 1 the fatigue driving detection system based on EEGs that integrates other features can realize the modernized and convenient function in the future.

In this literature review, a range of peer-reviewed survey from journals or conferences is completed. The steps involved in conducting this survey include the identification of resources and selection of papers, along with data extraction and synthesis. Recognizing the relevant keywords plays an important role in identifying the proper studies. An important and impartial survey, by searching for relevant keywords and filtering them, shows only articles written in English over the past decade. To select relevant studies, we excluded duplicates and ambiguous research questions, and assessed the accuracy and specificity of the remaining papers.

From the selected studies, we categorized the classification techniques used for driving detection relating to the EOGs as follows:

• Artificial neural network (ANN)

• Support vector machines (SVM)

• Fuzzy and neuro fuzzy based (NF)

• Clustering (CL)

• Inductive rule based (IR)

• Spectral bruit analysis (SBA)

• Bayesian learners (BL)

• Integration and differentiation based (ID)

• Extremely learning machine (ELM)

• Relevant vector machine (RVM)

• Principal component analysis (PCA)

• Independent component analysis (ICA)

As shown in Fig. 2 presenting the amount of researcher attention that each type of technique received during the last decade, an apparent publication peak is shown around the years 2019 and 2020. Overall, SVM, ANN, CL and NF are the four most frequently used ones; they together were adopted by 62% of the selected studies, as illustrated in Fig. 3. Figure 2 shows the distribution of research interest in each publication year. As can be seen, the activity of publications in this field is growing.

Fig. 2

Distribution of the studies over publication year

Fig. 3

Distribution of the classification techniques

Compared to other fatigue driving monitoring techniques based on EOGs, SVM, ANN, CL and NF seem to have received dominant attention in many years. In addition to these four classification methods, traditional methods such as ICA, PCA and IR have also been developed in recent years. In addition, the new methods, such as RVM and ELM, are also developing at an accelerated pace in recent years, showing good classification performance and practicality.

One of the main goals of this review is to pave the way for future researchers and practitioners by gaining insight into fatigue driving monitoring methods. To validate the stability of the developed machine learning model and verify its effectiveness, the researchers used performance measures to describe the reliability required for the evaluation. In terms of validation methods, cross validation is a common method. For example, reference 19 used the leave-one subject-out verification method, and the cross-validation accuracy of reference 10 was 80.74%. In addition to cross-validation, other performance measures are currently being used by researchers. An overview of the proportion of studies using each performance metric is illustrated in Fig. 4. According to the figure, it is found that Accuracy and correlation coefficient are popular performance metrics. 78.13% of the studies used Accuracy as a metric, while 12.5% used correlation coefficients. The most popular performance metrics was Accuracy, followed by correlation coefficient, root mean square error (RMSE), cross-validation, Recall, specificity, sensitivity, and the area under the receiver operating characteristic (ROC). Figure 5 shows the performance of the various classification algorithms in terms of Accuracy. According to the figure, in terms of Accuracy, ANN performed the best, followed by RVM, SVM and ELM, and NF and BL performed the worst.

Fig. 4

Proportion of various performance

Fig. 5

Accuracy of fatigue driving monitoring algorithm

The corresponding extraction algorithms and classification methods are different based on different signal types. In classification algorithms such as SVM, as a non-parametric method, the use of nonlinear discriminant function can overcome the limitation of parameter statistics [79]. SVM has outstanding capabilities of redundant feature processing, high-dimensional and small sample data processing. The ANN has been applauded for its excellent anti-noise ability, learning ability and adaptability. Although the current monitoring accuracy of SVM and ANN is high, the learning speed will decline when processing a large number of samples. Compared with other classifiers, ELM can greatly improve the learning speed [80]. ELM is also relatively robust to over fitting, outliers and noise [73].

Although the current research on fatigue driving is beginning to pay off, the performance of these methods is always different for different scenarios and different database tasks. Subsequent researchers will improve on previous studies to overcome their inherent shortcomings and form new research techniques for application.

Conclusion

Nowadays, with the rapid development of the transportation industry, traffic safety accidents related to the fatigue driving state occur frequently. In recent years, the research on relevant methods is also booming. However, different research methods, directions and algorithms vary in the accuracy of fatigue driving state detection. This paper summarizes and sorts out the current methods of fatigue driving state analysis and detection based on EOG signals, and also summarizes relevant feature classification methods. Fair and unbiased comments are given in this paper. In addition, this review does not include any work in progress, unpublished or non-peer reviewed publications. Finally, the strengths and weaknesses of fatigue driving detection techniques based on EOGs were extracted directly from the selected studies. This paper is a systematic literature review, 80 primary studies published during the last decade were identified. Although we cannot cover all relevant studies in this field, we believe it will be of some benefit to anyone who is interested in this field of research. The main findings obtained from the selected primary studies are:

• The classification techniques used for driving detection relating to the EOGs included Artificial Neural Network (ANN), Support Vector Machines (SVM), Fuzzy & Neuro Fuzzy based (NF), Clustering (CL), Inductive Rule Based (IR), Spectral Bruit Analysis (SBA), Bayesian Learners (BL), Integration and Differentiation based (ID), Extremely Learning Machine (ELM), Relevant Vector Machine (RVM), Principal Component Analysis (PCA) and Independent Component Analysis (ICA). Among them, SVM, ANN, CL and NF are used most frequently.

• Accuracy and correlation coefficient are the most commonly used for performance measurement in the primary studies. The overall estimation accuracy of most techniques is close to the acceptable level.

• ANN performed the best, followed by RVM, SVM and ELM, and NF and BL performed the worst in terms of accuracy.

Future researcher

The general trend is as follows: It is feasible to study driving fatigue by simulating test with driving simulator. On the basis of single feature monitoring, multi-feature fusion technology status monitoring will become the mainstream of research. Future technologies need to be improved from the following aspects:

• 1. Improve the real-time performance of the monitoring method. Traffic accidents happen in a very short time, which requires that the detection of driving fatigue can be rapid, accurate and timely, and can give early warning to drivers. However, the existing driving fatigue detection methods are poor in real time so far.

• 2. Eliminate or significantly reduce the intrusion of the detection system. The method based on EOGs has a high accuracy, because it can monitor the physiological state from the human body in real time, but it may affect the driver's operation. Low contact is a major challenge for future hardware facilities.

• 3. Improve the cost performance of relevant technologies. Even though EOG is cheaper than other methods, the equipment required by existing conditions is still complex and expensive, which makes it difficult to popularize. The post-processing data of multi-feature fusion technology is huge and the algorithm is complex, so simplifying the algorithm and controlling the cost is a great challenge in the future.

• 4. Establish a set of accurate and recognized fatigue classification criteria. The current fatigue detection technology has no mature reference to accurately classify the fatigue detection level.

• 5. The development of relevant technologies should consider different driving environments and road conditions, and develop relevant fatigue detection technologies suitable for different professional drivers.

Ideally, reliable fatigue detection equipment in the future can not only accurately identify drivers’ fatigue, then warn them, but also notify the third party to intervene when necessary, which will greatly improve the safety of transportation.