Multibiometric identification by using ear, face, and thermal face

Abstract

In this work, a secure multibiometric system is proposed. Three different biometric modalities which are ear, face, and thermal face are considered. The face and thermal face data were taken from USTC NVIE Spontaneous Database, whereas the ear data were collected from IIT Delhi Ear Image Database. For each modality, three feature extraction methods are used and four different classifiers (multilayer perceptron, decision tree, support vector machines, and probabilistic neural network) are trained by using two fusion methods which are matching score level and feature level fusion. According to the results, the individual biometrics are better for the identification problem. However, for the validation problem, both fusion methods give better false acceptance rate/false rejection rate values regarding to individual biometrics.

1 Introduction

In today’s conditions, security is an essential concept for many domains such as online or mobile banking and controlled access to certain buildings or rooms. Biometric identification and verification offers a more secure solution for such problems; however, in real applications, single biometrics does not guarantee the ultimate safety [1].

In certain conditions, biometric systems may be tricked by using forged biometric samples such as photographs of faces or irises and artificial fingerprints. Another disadvantage is that, if a biometric system uses only one kind of biometric measurement, it may not be readable or reachable due to physical conditions. For these reasons, the use of multibiometric systems has come to the forefront [2].

In general, systems using biometric features are more secure than encryption systems using passwords, keys, or keycards [3, 4]. Stealing a password or a key is relatively easier than copying a biometric sample because a biometric sample is a part of the subject who uses it. On the other hand, a biometric authentication system is not 100% secure. A skillful attacker may gather any person’s biometric samples and can produce forged biometrics.

A biometric sample must be universal (most of the population should have it), unique, stable (constant over time), collectible (easy to collect with proper devices), distinct (able to differentiate people from each other), acceptable (should be acceptable for people), and cheap. Table 1 compares the most frequent biometrics regarding to these features.

Table 1 Biometrics and parameters (5 high–1 low)

In general, the aim of this paper is to develop a touchless biometric system with a high accuracy and low cost. To achieve these goals, three biometrics, which are ear, face, and thermal face, are selected from Table 1. Regarding their scores, these biometrics are better fitting to our problem [5,6,7].

The organization of the paper as follows. Section 1 proposed the multimodality concept. Section 2 summarizes the methods involved in this work. The experimental details and the results are proposed in Section 3, and Section 4 concludes the paper.

1.1 Multimodalities

Multibiometric systems have many advantages over single-biometric systems: First of all, a multibiometric system can work well against class similarities. A multibiometric system is more robust to noise and more resistant to external system attacks [8, 9]. The basic definitions of multibiometric systems are as follows. A multimodal system is that receive two or more different biometric characteristics as input with the aid of one or more sensors [10]. A single biometric characterization process using a single sensor is called as multi-algorithm. A multi-algorithm system is a classification process performed by using two or more algorithms [11]. If a system captures more than one example for a single biometric characteristic and uses these multiple copies for classification or identification by using the same algorithm, it is called as multi-instance [7]. If the system has more than one sensor to capture the biometrics, it is called as multi-sensor system [12].

1.1.1 Fusion of multimodalities

A typical biometric system has a sensor module, feature extraction module, matcher, and decision module. The information on the output of these modules can be combined and used as an input to other modules.

1.1.2 Sensor-level fusion

The purpose of sensor-level fusion is to fuse the same biometric samples coming from different sensors or multiple instances captured by the same sensor [13]. This method gives more information about the biometric data, but it is still dependent on only one biometric. Figure 1 shows the block diagram of sensor-level fusion.

Fig. 1

Block diagram of sensor-level fusion

1.1.3 Feature-level fusion

The feature-level fusion contains the richest information after the sensor-level coalescence. In this method, different feature extraction algorithms are applied to the same biometric data. Using different feature extraction methods causes a large amount of data; hence, a feature reduction scheme should be necessary to reduce the feature size because of the curse of dimensionality. Figure 2 shows the block diagram of feature-level fusion.

Fig. 2

Block diagram of feature-level fusion

1.1.4 Matching score-level fusion

The matching score is generated by a metric based on the similarities between the example in the input of the biometric system and the training samples. If more than one metrics are used, it is possible to fuse their scores [14]. The output values of the metric functions are simple numbers; hence, fusing the matching scores is much easier than the other fusing methods. If the confidence levels of the classifiers are not equal, a normalization of scores is required before fusion [15]. Figure 3 shows the block diagram of matching score-level fusion.

Fig. 3

Block diagram of matching score-level fusion

1.1.5 Decision-level fusion

Decision-level fusion uses outputs of different classifiers to construct the final decision [16]. In this method, each classifier can use the same biometric data as well as different biometrics and/or features. Similar to the matching score-level fusion, if the confidence levels of classifiers are different, it is possible to use a weighted decision scheme [17]. Figure 4 shows the block diagram of decision-level fusion.

Fig. 4

Block diagram of decision-level fusion

2 Methods

In this section, the feature extraction, the feature selection, and the classification methods used in this study are explained briefly.

2.1 Feature extraction methods

In this study, three different methods which are eigenfeatures, local binary pattern (LBP) features, and gray-level co-occurrence matrix (GLCM) features were used to extract the features from data.

Mathematically, the principal components of a given set of biometric images characterize the variations of the data [18, 19]. In other words, any image in the set is expressed as a linear combination of principal components. The number of possible principal components is equal to the number of the images in the set. However, the contribution of some principal components is small enough and negligible. The most dominant components are related to the greatest eigenvalues of the covariance matrix of the image set. Then, the problem reduces to find the eigenvectors of the covariance matrix related to greatest M eigenvalues.

LBP is a fast feature extraction method for images. In this method, a threshold operator is applied to 8-neighborhood of a selected pixel of the image. The center pixel’s value is used as threshold, and one bit is produced for each neighbor. If the neighbor’s value is larger than the threshold, its value is set to 1, or 0 otherwise. An eight-bit codeword is generated with these eight bits [20].

GLCM is a statistics-based feature extraction method. In this method, repeated gray level patterns are counted and stored in a matrix. For example, if only one direction is considered, an eight-bit gray-level image produces a GLCM size of 256 × 256, which is as large as the original data. Hence, we calculated four features over GLCM which are contrast, correlation, energy, and homogeneity, instead of using the whole matrix.

2.2 Feature selection method

In general, feature selection algorithms are used to reduce the amount of data and computational cost. On the other hand, it is possible to use a feature selection algorithm to remove the redundant features and to improve the overall accuracy of the problem. In this study a forward selection method is used on the features. The algorithm starts with an empty set of features. Then, the existing features are applied to the classification process one by one. The best resultant feature is determined and added to the feature set and another new feature is determined with the same rule. The selection process is repeated until the increase of success rate is stopped or the maximum allowed number of features is reached.

2.3 Classification methods

In this study four different classifiers, which are multilayer perceptron, decision tree, support vector machine and probabilistic neural network [19, 21,22,23], are used for classification. Those methods are used on different levels of fusion process. The success rates as a result of classification operations are given in the experimental results section.

3 Experimental results and discussion

In this study, three different biometric measurements which face, ear and thermal face, are considered to design a secure biometric system. For each modality, three different feature sets were calculated and four different classifiers were utilized. Thus, we tried matching score-level and feature-level fusion to obtain the best configuration.

The face and thermal face data were taken from USTC NVIE Spontaneous Database [24,25,26], whereas the ear data were collected from IIT Delhi Ear Image Database [27, 28]. The dataset was created artificially by combining these two datasets because it is not possible to find a proper dataset for our work. In our dataset, there are 120 individuals. Each person was represented by ten face and ten thermal face images with spatial resolutions of 640 × 480 and 304 × 230 respectively. We added three ear images to each person artificially. Hence, our dataset consists of 2760 biometric measurements assigned to 120 individuals.

Since the eigenfeatures come from principal component analysis and selecting lesser eigenvectors reduces the size of data, there is no need to use any feature selection process on this part of the data. In the first stage of the experiments, only the eigenfeatures were considered. To find better accuracy, different sizes of eigenfeatures were calculated and processed by classifiers. Then, the experiments were repeated by using LBP and GLCM. For these two feature sets, the most distinctive features were determined by using forward selection. All experiments were repeated ten times by using randomly selected training and test sets, and the results were averaged.

In the first step, data were divided into training (70%) and testing (30%) sets randomly for different numbers of eigenfeatures. Then, one multilayer perceptron (MLP) was trained for each biometric to obtain a matching score. In the matching score-level fusion, output scores of the classifiers were combined by using a genetic algorithm (GA). Tables 2, 3, and 4 summarize the results for this step. In this step, the optimum hidden layer size of MLP was found by a trial by error scheme (Tables 2, 3, and 4). Table 5 shows the feature-level fusion scores by using eigenfeatures and MLP. In this step, equal numbers of eigenfeatures were calculated for each biometric and then merged to construct the dataset. A single MLP fed by these data for training and testing.

Table 2 Matching score-level fusion with eigenfeatures (#feature 30)

Table 3 Matching score-level fusion with eigenfeatures (#feature 40)

Table 4 Matching score-level fusion with eigenfeatures (#feature 50)

Table 5 Feature-level fusion with eigenfeatures (#feature 30–40–50)

Secondly, LBP and gray level co-occurrence matrix features were combined. Decision tree (DT) and linear support vector machines (SVM) and probabilistic neural network (PNN) algorithms were used as classifiers. Similar to the first step of the experiments, both matching level and feature-level fusions operated on the data. To increase the accuracy of the system, a forward selection process was applied for each classification.

Tables 6, 7, and 8 show the matching score-level fusion results for three classifiers.

Table 6 Matching score-level fusion with DT

Table 7 Matching score-level fusion with SVM

Table 8 Matching score-level fusion with PNN

In the last part of the study, we evaluated feature-level fusion by using three classifiers. Tables 9 and 10 summarize the result of feature-level fusion experiments.

Table 9 Feature-level fusion with DT

Table 10 Feature-level fusion with PNN

Finally, we evaluated the whole problem as an identification task. In this last step, the classifiers were trained for every individual separately to obtain the false acceptance rate (FAR) and false rejection rate (FRR). FAR, FRR, and testing times of each biometry and fusion types are given in Table 11 [8]. All the experiments were realized, and a personnel computer has a processor of Intel Core i5-5200U 2.20 Ghz and 8Gb of RAM.

Table 11 FAR, FRR, and testing time results for each method used in the experiments

According to the results given above, ear is the most decisive biometric among the others. By using LCM + GLCM, SVM classified the entire data perfectly. By comparing Tables 2, 3, 4, and 5, one should claim that feature-level fusing gives slightly better results than matching score-level fusion for eigenfeatures and MLP. When LBP and GLCM are used as features, DT and PNN produces similar results. However, when SVM is considered, matching score fusion produced a better result than feature-level fusion.

When Table 11 is interpreted, FAR and FRR rates of both fusion methods are better than simple biometrics. MLP, DT, and PNN produced slightly better FAR/FRR values with feature-level fusion, whereas SVM gave better results with matching score level. However, the differences between the fusion methods are very small; hence, it is not easy to prefer one to other.

4 Conclusions

In this work, a multibiometric system is proposed. Three different biometric modalities were considered. For each modality, three feature extraction methods were used and four different classifiers were trained. Two fusion methods were utilized, and the results were discussed. According to our results, if the problem is considered as an identification problem, using the ear only gives the best result. For the identification problem, none of the considered fusion methods improved the results. However, when it is interpreted as a validation problem, both fusion methods gave better FAR/FRR values regarding to individual biometrics. SVM with matching level fusion also gave the best classification performance as 98.65%.