The use of hidden Markov models to verify the identity based on facial asymmetry

Abstract

This work concerns the use of biometric features, resulting from the look of a face, for the verification purposes. Different methods of selection and feature analysis during face recognition are presented here. The description contains mainly analysis possibilities and also identity verification based on asymmetric facial features—in later stages. The new verification method has been introduced based on designated characteristic points. These points were designated through appropriate coded information about facial asymmetry as observation vectors and recognition using hidden Markov models. The advantage of these models is that we can use vectors of different lengths. Such vectors appear in incomplete data when all points cannot be located.

1 Introduction

There are many highly effective methods of identification and user verification based on biometric features. There can be distinguished features fetched base on the iris image and features contained in a hand. All of them require some user-system cooperation or also a person measures it. A non-invasive example of biometric system can be a system which executes the verification and identification based on facial features.

The systems of face recognition (FR) are used in systems requiring data security mainly because of mentioned non-invasiveness [1]. A user does not usually take part in the process of data consumption and often does not need to know about that [2]. The face recognition means can be defined as searching of all faces in a still image or a video sequence. The identification or the authentication can be done by comparing found faces with those in the database [3].

The research concentrates on construction of fully automatic face recognition system due to the possibility of common and cheap use of computer systems [4]. That face recognition system combines methods of face localization and extraction of structural characteristics of a face, such as eyes and nose shape and size, face shape, face colour and other distinguishable features and holistic methods, such as eigenfaces or fisherfaces. Using face structural features, there have been searched new identification and identity verification methods. Dependencies used as the main recognition factor are imperceptible at first [5–7]. There are many works on face recognition that use the asymmetry of the face, and the efficiency of this technique is confirmed in articles [8–10]. These methods used differences in geometrical features of the face. The important thing during extraction of these features is the choice of asymmetry line, and there are few ways to do this. Once the popular base on centre of the line connected inner eyes corner [11, 12]. Next, the asymmetry measurement is calculated and used for identification. Few papers in the literature are dedicated to the 3D asymmetry face recognition task so far. Huang et al. [13] propose method based on local binary pattern (LBP). Their approach splits the face recognition task into two steps: (1) a matching step respectively processed in 2D/2D; (2) 3D/2D a fusion step combining two matching scores. Canonical correlation analysis (CCA) is applied in method proposed by Yang et al. [14]. They apply CCA to learn the mapping between the 2D face image and 3D face data.

The works about the analysis of facial asymmetry were based, most of the time, on feature vectors construction received from the image in automatic or semi-automatic way. They were compared by one of the known distant method. However, there is the risk of getting wrong results due to determining characteristic face points in an inaccurate way. The introduction of some additional determinants can be useful, and the appropriate coding of these determinants allows using hidden Markov models (HMM) for recognition. HMM work using vectors with different length. The stability of the verification system is maintained even when some features in the process of observation vectors creation (e.g., by the error because of localization of face characteristic points) are skipped.

2 Methods

2.1 Automatic face detection

The sine qua non for automatically working of the method based on facial asymmetric features is executing automatic face detection in static image or even a frame sequence. It is assumed that main used data is static data, so the video sequence should be analysed frame by frame as a static image [15, 16].

In the case of face recognition based on whole look, provided data does not have to have high resolution and very good quality. It is known that a man can recognize faces and identify them correctly even in the case of blurry images, out of focus and also small sizes. However, the analysis of facial asymmetry is based on certain features which may be defined simply only in images with high resolution and focus. It is also important that a face is arranged frontally with open eyelids and eyes looking straight. The method based on specific skin colour is used to locate the face. It is used with regard to a base of photos taken in similar lighting conditions and components of used methods using information about colour [17, 18].

There follows the detection of facial characteristic points – after appointing the area face area. These points are chosen as key elements of determining asymmetry method [15]. There are following characteristic points: corners of both eyes—internal and external; centres of circles describing the iris; mouth corners; points describing the width of the nose in its widest place. Figure 1 shows the scheme of searching face areas in order to find the characteristic points.

Fig. 1

The scheme of searching face areas in order to find the characteristic points

2.2 Detection of characteristic points of the mouth

One of the most popular mouth detection methods is presented by Viola and Jones [19, 20]. They presented a robustness algorithm combined of Haar-like feature, Adaboost algorithm and Cascade method to filter and find the correct face region. The other method is proposed by Huang and Lin [21]. This approach concludes preprocess, segment of the face region, edge detection and projection. The skin colour segmentation is used to find face mask which purposes to filter out non-skin colour regions and raises accuracy before mouth localization process. Similar method is proposed by Khan et al. [22]. They proposed a hybrid method based on Viola-Jones and skin colour detectors.

The method based on the specific colour of the mouth is accepted in order to determine correctly its corners. This method assumes the determination of pixels which fulfil threshold values, assorted experimentally. It takes place in a calibration process. The way of the operation on RGB components, which detects mouth colour, is described in detail in the work [15]. If you know the mouth shape, you can designate its corners to the utmost in non-zero pixel level. Searching such pixels need to be executed in limited area of the mouth, near horizontal axis.

2.3 Detection of eye area and eye characteristic points

In the case of the eyes, the localization takes place by the other checked method. It is based on searching face elements which have significant value variations in the level of pixel brightness in small areas (so-called gradient method) [23–25]. There may be observed significant value variations in the eye area where the white of the eye is similar to the maximum white colour and the pupil to the maximum black colour, although the iris may contain also white light reflections.

After determining eye localization, one goes to the next stage. It requires precise designation of eye corners and centre of circle describing the iris, which was described in detail in work [15]. As it will be shown, the iris has a very significant importance in this method. A man has constant diameter of the iris, from birth till the old age. It amounts 10 ± 0.05 mm. It is an essential information: based on constant size of any man’s iris diameter, any photo may be scaled to the same real size. A measurement of facial asymmetry in proposed method should be carried out on actual dimensions. The iris size is determined as a unit ‘Muld’ [26–29].

The localization of eye corners requires determining the curves describing the lower and upper edges of the eyelid. The point of curve intersection is designated by the eye corners. The detection of the eyelid edges may be executed based on modified parabolic Hough transform. The eyelid edges may be designated with methods using information about level luminance of analysed area, the method analysing image depth in greyscale [30]. Figure 2 shows the image of eye area in dimensional space, where the values of certain pixels are presented by brightness levels.

Fig. 2

Three-dimensional image of the eye area, presented in different levels of brightness

In order to determine approximate shape of the eyelids, it is necessary to fix appropriate threshold value. That enables to define what constitutes the eyelid edge in the image and what is only, e.g. disrupt wrinkle. However, conducted analyses of luminance levels in the image limited to eye area shows that proposed method works quite precisely but only for the upper eyelid. The lower eyelid often is not clear enough to determine its edge, maintaining big range of threshold values. Small range of these values causes unfortunately too low versatility of the method and the necessity of constant calibration.

Similar to detection of mouth edges, in the case of the eye (similar to detection of mouth edges), if one determines approximately the eye corners at the beginning, then it is easier to search the eyelid edges. There should be sought only limited sub-areas to narrow down the searched area. Limited sub-areas consist of matrix elements [31]:

$$ \left\{ \begin{array}{ll} \text{Iris}\_y~-~\text{Iris}\_x~\leq~j~\leq~\text{Iris}\_y~+~\text{Iris}\_r\\ \text{Iris}\_y~-~P~\leq~i~\leq~\text{Iris}\_y~+~P \end{array} \right. $$

(1)

$$ P=\text{Iris}\_x-\sqrt{\text{Iris}\_{r}^{2}-(j-\text{Iris}\_{y}^{2})} $$

(2)

where: P is the limited search area, Iris_x, Iris_y are the coordinates of the citcle centre defining the iris, Iris_r is the radius of the circle defining the iris, and i,j are the column number and row number in the luminance matrix, respectively. See Fig. 3.

Fig. 3

Illustration for Eq. 1 and 2

The leftmost point of the lowest luminance value is assumed for the outer eye corner, and the rightmost point of the least luminance is assumed for the inner eye corner. Additionally, the distance from the centre of circle describing the iris is taken into consideration. Similar assumptions are taken for the right eye. Searching of the upper eyelid edge takes place by moving from one eye corner to the second along the accepted arc. What is more, the column of the luminance matrix is searched in range <Y p − b,Y p + 2b>, where Yp is the extreme coordinate gained by searching the previous column (for the first column, the coordinate of the eye corner is assumed) [31]. In such a way, approximate localization of the eye corner edges is designated. N points are received, where n determines the assumed number of columns between the corners of the eye. The example of rough detection of eye corners and upper edge of the eyelids is shown in Fig. 4.

Fig. 4

The example of rough detection of eye corners and upper edge of the eyelids

Too low difference of luminance value in the case of the lower eyelid causes the necessity to use another detection procedure of its edge. The procedure which uses information about gradient luminance in specified intervals (image lumination procedure (ILP)) in connection with the analysis of components in (R/G, B/G) space [31] may be an interesting approach. In ILP procedure, one has to divide the eye area into five intervals, 30 columns each, to execute the project of luminance gradient value within each range according to the following equation:

$$ \textrm{ILP}(y)=\sum\limits_{KZ_{y}-1.5*Iris\_r}^{KZ_{y}}|I_{\text{lum}}(S_{i}+x)(y)-I_{\text{lum}}(S_{i}+x+1)(y)| $$

(3)

where: I _lum is an array of luminance values, K Z _y is a vertical coordinate of the outer corner, and S _i is the initial value of the ith interval (i = 1 … 5).

The line containing the point of the lower eyelid edge in a given interval is determined by the biggest value in ILP table, and the column is determined by half of a given interval. In such a way, five points are gained. There may be approximated the shape of the lower eyelid edge for these points.

The information from the space may be used to determine the lower edge of the eyelid if the analysed image contains the information about the colour. An example of such an approach is that one used in the work [32] for the detection of lip contour. The image from RGB space is converted into (R/G, B/G) space according to the following dependence [31]:

$$ \left\{ \begin{array}{ll} I_{RG}(y)(y)=I_{R}(x)(y)/I_{G}(x)(y)\\ I_{BG}(y)(y)=I_{B}(x)(y)/I_{G}(x)(y) \end{array}\textrm {for {{x}~=~0\ldots,{w,y} = 0\ldots {h}}} \right. $$

(4)

where w and h are the width and height of the input image in pixels, respectively; I _R , I _G , I _B are the R, G and B components in the RGB colour space, respectively; and I _RG and I _BG are the R/G and B/G components in the representation of the image in (R/G, B/G) space.

Depending on accepted threshold values, the analysis of the components in (R/G, B/G) space allows for accurate separation of definite colour area. That searched area is determined according to the following formula:

$$ I(x)(y)= \left\{ \begin{array}{ll} 1 & \text{for}~ {I}_{RG}({x})({y})<{{T}_{1}}~ \text{and}~ {I}_{BG}({x})({y})>{T}_{2}\\ 0 & \text{otherwise} \end{array} \right. $$

(5)

for x=0...w,y=0...h, where w and h are the width and height of the input image in pixels, respectively; I _RG and I _BG are the R/G and B/G components in the representation of the image in (R/G, B/G) space, respectively; I is the searched area; and T ₁ and T ₂ are the threshold values.

In addition to Eq. 5, the assumption may be added that pixel values for green component G should be in the range above the threshold T ₃. All threshold values should be chosen in the experimental way. In such systems, the selection of threshold values is often in the system calibration process during the work in a particular environment. The example of searched area detection in order to determine the lower eyelid edge and correction of eye corner localization is illustrated in Fig. 5.

Fig. 5

Detection of the white of the eye area in the designation of the lower eyelid edge

The localization of designated five points using the ILP procedure is corrected with the edge defined by thresholding in a (R/G,B/G) space.

Some points on the edges of the upper and lower eyelids may be determined in a wrong way. Approximation of designated points may be executed using polynomial of third degree in order to level the differences between the real and originally designated eyelid edge localization. In addition, approximating curves are used to determine final localization of the eye corners. These curve intersection—near the designated earlier corners—determines their proper localization.

Polynomials approximating the shape of the eyelid edges may be designated using the method of smallest squares [10] aiming at minimization of χ ² size:

$$ \chi^{2}=\sum\limits_{i=1}^{n}\frac{(y_{i}-f(x_{i}))^{2}}{\sigma_{i}^{2}} $$

(6)

where: y _i ,f(x _i ) is the expected value and the value of the function for the ith point, $\sigma _{i}^{2}$ is the standard deviation for the ith point and n is the number of points.

Because of less accuracy of designating points which describe the eyelid edges near eye corners, the value of standard deviation is assumed individually for each point. In such a way, variables of weight values are determined depending on the localization of points between the distance determined by corners. The presented procedure concerning detection of the edges of the upper and lower eyelid is an essential element of functioning the whole method of identity verification based on facial asymmetry. The eyelid edges are not used directly, but they are only one element of procedure of eye edge localization. Correctness of designating the curves which approximate the eyelid edges has large meaning in eye edge appointing. Therefore, in real conditions, one has to put a lot of attention to the correct calibration by the best threshold values selection. The example of appointing lower and upper eyelid edges and the circle describing the iris and corners is shown in Fig. 6. Searched characteristic points are determined by blue crosses.

Fig. 6

Example of designation searched points and edges, under the eye

2.4 Determination of the characteristic points of the nose

The last of searched characteristic points are about the nose’s width in its widest place. It is one of the most difficult tasks which should be conducted in the proposed method of facial asymmetry measuring. In the case of while searching chosen points in the eye or mouth area, analysed areas are characterized by large contrast between certain elements in the level of colour or greyscale. Such big differences in contrast scale are not observed in the nose area. That is why the depth of the luminance level is taken into consideration. The difficulty of analysis results from the fact that the searched nose area changes its position in various people. There is no ideal way to identify clearly which area should be searched. The proposed scheme of determining area assumes that the area would be designated by localized centres of the eyes (the width d) and also by the vertical distance from 0.8 k to 0.8 d, where d is the distance between the centres of the eyes and k is the distance between the inner corners of the eyes (see Fig. 1). Such division results from taken tests based on available pictures. It happens, however, that among some people, the proposed scheme omits searched points of the nose, so the calibration of system settings is necessary, dedicated to a user.

After correct designation of the area containing the nasal element in its widest place, the further work is about area analysis with additional assumptions about searched point localization. It is known that the points will be on both sides in relation to the image vertical axis of symmetry. To analyse the image depth at the level of luminance, searched image is being converted into the greyscale (similar to the case of determining the upper eyelid edges). In the next stage of image searching, one would find more significant differences in the luminance depth, moving horizontally from the edge of the image to its axis of symmetry. Such differences are probable to find in a good quality picture, without X-rays and unnecessary shadows on the face. After searching all lines among designated points, the most extreme point on symmetry axis’ left and right is chosen. The designated points define nose’s limits in its widest place. The example of designating characteristic nose points is illustrated in Fig. 7.

Fig. 7

Example of determining specific characteristic points of the nose

The beard affects on the method’s efficient functioning, as it is demonstrated on the above figure. Even better, it is illustrated by the three-dimensional visualization in Fig. 8.

Fig. 8

Spatial visualization of the depth luminance for the area of the nose

To avoid the beard disturbing influence, one may make following reservation in the method of seeking nose characteristic points: the relevant points are being determined on the basis of all located points, arranged in a pattern (vertical curves), resembling the shape of nose piece. All points which significantly differ from the vertical arc shape should be removed. One attempts to find blowholes or other points on the basis of analysed pictures, but not always and not in every pictures they may be located. The important information may be also the point with the highest luminance level. It is almost always placed in the central nose part. Points determining the nose width may be compared with the nose central point. The final localization may be determined on this basis.

2.5 Create a template of the facial asymmetry

Based on fixed and accepted characteristic face points, special template of asymmetry is built and then, in a later stage, the construction of facial asymmetry observation vector [7]. Facial asymmetry is measurable if every time and for all face analysed pictures, there are fixed the same work conditions and all algorithms and methods of characteristic points determining give correct answers. If points’ coordinates are in chronological feature vector, then the verification or even the identification leads to comparison. of the distance between two feature vectors. In real situation, working with the fully automated system, there may be observed various types of obstacles impacting, more or less, on the correct working of the system. It is necessary to find better solution, which allows to level the influence of distracting external factors on efficient operation system. This paper describes the identity verification method based on facial asymmetry using hidden Markov models. Hidden Markov models may generalize in a controlled way which comparing the observation vector amounts to find many similarities between compared vector and the vector which a given model was taught. In practice, it means that the models would work correctly despite disturbed correctness of feature vector. This property was used in the case of proposed method of identity verification based on asymmetric feature vector. Determining characteristic points, every time in a simple way, is not often attainable. However, if a certain margin of error is taken, the possible functioning of the method would be probable in spite of some discrepancies.

The characteristic points fixed in earlier stages would be used to create the template of facial asymmetry. These points are located in the eye area (internal and external corners, the centres of circles describing the iris), in the nose area (extreme points describing nose’s maximum width) and the mouth area (mouth corners). Template creation begins with determining the facial symmetry axis. Proposals to determine perfect symmetry axis one may find are in various scientific studies; however, the most reliable axis was determined in the works [27, 29]. A little modified version was accepted in this work. Mainly, the face symmetry axis is determined based on two points. First of them, A, is the half of segment determined by the inner eye corners. The second one, B, is the half of segment determined by the mouth corners (Fig. 9). Based on face symmetry axis, the following elements of asymmetry template are determined.

Fig. 9

Diagram of template creation of facial asymmetry

The observation vector of facial asymmetry is built based on determined distances, angles and points of intersection of auxiliary lines. There shown in Fig. 10 a fragment of the facial asymmetry template with determined distance values.

Fig. 10

Fragment of template of facial asymmetry with the designated distance values

Designated distances:

• x1 - diameter of the circle defining the outer edge of the right eye iris (≈1 Muld),

• x2 - diameter of the circle defining the outer edge of the left eye iris (≈1 Muld),

• x3 - the distance between the inner and the outer corner of the right eye

• x4 - the distance between the inner and the outer corner of the left eye

• x5 - the distance between the outer corner of the right eye and the right corner of the mouth

• x6 - the distance between the outer corner of the left eye and the left corner of the mouth

• x7 - the distance between the outer corner of the right eye and the right point of defining the width of the nose

• x8 - the distance between the outer corner of the left eye and the left point of defining the width of the nose

• x9 - the distance between the right point of defining the width of the nose and the right corner of the mouth

• x10 - the distance between the left point of defining the width of the nose and the left corner of the mouth

• x11 - the distance between the inner corner of the right eye and the right corner of the mouth

• x12 - the distance between the inner corner of the left eye and the left corner of the mouth

• x13 - the distance between the corners of the mouth

• x14 - the distance between the inner corners of eyes (distance k)

• x15 - the distance between the inner corner of the right eye and the center of a circle circumscribing the outer edge of the iris of right eye

• x16 - the distance between the inner corner of the left eye and the center of a circle circumscribing the outer edge of the iris of left eye

Designates angles:

• α1 - the angle between the line defined by the outer and inner corners of the right eye, and the line set by the inner corners of the eyes

• α2 - the angle between the line defined by the outer and inner corners of the left eye, and the line set by the inner corners of the eyes

• α3 - angle between the line defined by the corners of the mouth, and the line perpendicular to the axis of asymmetry of the face, passing through the right corner of the mouth

• α4 - angle between the line defined by the corners of the mouth, and the line perpendicular to the axis of asymmetry of the face, passing through the left corner of the mouth

• α5 - angle between the line defined by the points defining the width of the nose, and the line perpendicular to the axis of asymmetry of the face, passing through the right point indicating the width of the nose

• α6 - angle between the line defined by the points defining the width of the nose, and the line perpendicular to the axis of asymmetry of the face, passing through the left point indicating the width of the nose

There has been assumed the ruIe for computing the ratio between suitable values for the face’s right and left part relative to the symmetry axis – in order to code all values describing facial asymmetry using the distance and the angles:

• D1 = x1/x2,

• D2 = x3/x4,

• D3 = x5/x6,

• D4 = x7/x8,

• D5 = x9/x10,

• D6 = x11/x12,

• D7 = x13/x14,

• D8 = x15/x16,

• D9 = α1/α2,

• D10 = α3/α4,

• D11 = α5/α6,

It has been assumed that the ratio of corresponding values ranges from 0.5 to 2. In extreme cases, some dimensions would be twice bigger than the same dimensions for the other parts of the face. Imputing of observation symbols takes place by dividing accepted range (from 0.5 to 2) into sub-bands and accepting one unique observation symbol for each sub-band. In such a way, there are number symbols 1, 2, …, 20, for values from 0.5 to 1, and there are number symbols 21, 22, …, 40, for values from 1 to 2. The first 11 observations in facial asymmetry vector result directly from value of encoding for $D 1, D 2, \dots, D 11$.

For the sake of greater work stability resulting from the accuracy of determining certain face characteristic points, obtained observation vector was supplemented by further observations. The observations resulting from the the straight line intersection, as depicted in Fig. 9, are determined by:

• The outer corner of the right eye and the right corner of the mouth, and the outer corner of the left eye and the left corner of the mouth, in the point W

• The right corner of the mouth and the right point designated width of the nose, and the left corner of the mouth, and left points of the designated width of the nose, in the point X

• The outer corner of the right eye and the right point designated width of the nose, and the outer corner of the left eye and the left point designated width of the nose, in the point Y

• The right corner of the mouth and the inner corner of the right eye, and the left corner of the mouth and the inner corner of the left eye, in point Z.

Points W, X, Y and Z are located in the areas covered by special network (6 from 10 cells).In that network, one cell, for points W and Z, has size of 0.5 Muld, whereas one cell, for points X and Y, has size 0.25 Muld. On Fig. 11, four grids are presented with points of intersection determined by the straight line.

Fig. 11

Four template grids of facial asymmetry for encoding the last four observations

All networks are arranged in parallel and symmetric way to the face axis. The network for point W has its upper position 11 Muld below point A, the network for point X has its lower position 2 Muld below point A. The network for point Y has its upper position 4 Muld below point A and the network for point Z has its lower position 4 Muld above point A (see Fig. 9). Each network contains observation symbols ranging from 1 to 60. If the intersection point of the straight lines is beyond the network area (areas 0, 1, 2 or 3), given feature would be omitted (Fig. 11). The network location was preceded by a number of studies of different face types, and they are appropriate almost for all accepted positions. It may happen that due to wrongly taken picture (poor lighting, poor focus), the system locates the characteristic points imprecisely and the next stages would be affected by the initial error.

2.6 The use of hidden Markov models

The hidden Markov model (HMM) is a powerful statistical tool for modelling generative sequences that may be characterized by an underlying process generating an observable sequence. HMMs have found application in many areas interested in signal processing and, in particular, speech processing but have also been applied with success to low level natural language processing tasks such as part of speech tagging, phrase chunking and extracting target information from documents Fig. 12. HMMs are stochastic models and are widely used for characterizing the spectral properties of patterns’ frames [33–38].

Fig. 12

ROC

A HMM is characterized by the number of states in the model N, the number of observations M, the state transition probability distribution A, the observation symbol probability distribution B and the initial state distribution π. The compact notation λ = (A,B,π) is used to indicate the complete parameter set of the HMM model.

HMMs have three fundamental problems: recognition, segmentation and trying. These problems may be defined as follows:

• Recognition problem is computing the probability P(O|λ) given the observation sequence O and the model λ

• Segmentation problem is the determination of the optimal state sequence given the observation sequence O = O1,O2,…,O T, and the model λ

• Training problem is the adjustment of model parameters λ = (A,B,π) so as to best account for the model states, this is equal to adjust the model parameters λ = (A,B,π) to maximize P(O|λ).

Given a HMM, and a sequence of observations, we would like to be able to compute P(O|λ), the probability of the observation sequence for given model. One may view this problem as one of evaluating how well a model predicts a given observation sequence and thus allow us to choose the most appropriate model from a set. The probability of the observations O for a specific state sequence Q is:

$$ P(O|Q,\lambda)=\prod\limits_{t=1}^{T} P(o_{t}|q_{t},\lambda)=b_{q_{1}}(o_{1})b_{q_{2}}(o_{2})...b_{q_{T}}(o_{T}) $$

(7)

and the probability of the state sequence is:

$$ P(Q|\lambda)=\pi_{q_{1}}a_{q_{1}q_{2}}a_{q_{2}q_{3}}...a_{q_{T-1}q_{T}} $$

(8)

so we can calculate the probability of the observations given the model as:

$$ P(Q|\lambda)=\sum\limits_{Q} P(O|Q,\lambda)P(Q|\lambda)= $$

(9)

$$\begin{array}{@{}rcl@{}} =\sum\limits_{q_{1}...q_{T}} \pi_{q_{1}}b_{q_{1}}(o_{1})a_{q_{1}q_{2}}b_{q_{2}}(o_{2})...a_{q_{T-1}q_{T}}b_{q_{T}}(o_{T}) \end{array} $$

This result allows the probability of O evaluation, but to evaluate it directly would be exponential in T.

A better approach is to recognise that many redundant calculations would be made by directly evaluating Eq. 9, and therefore, caching calculations may lead to reduced complexity. We implement the cache as states trellis at each time step, calculating the cached valued (called α) for each state as a sum over all states at the previous time step. α is the probability of the partial observation sequence o ₁,o ₂,…,o _t and state s _i at time t. We define the forward probability variable:

$$ \alpha_{t}(i)=P(o_{1}o_{2} \ldots o_{t}, q_{t}=s_{i}|\lambda) $$

(10)

so if we work through the trellis filling in the values of α, the sum of the trellis final column will equal the probability of the observation sequence. The algorithm for this process is called the forward algorithm and is as follows:

Initialization:

$$ \alpha_{1}(i)=\pi_{i}b_{i}(o_{1}),~~~~~1\leq i\leq N $$

(11)

Induction:

$$ \alpha_{t+1}(j)=\left[\sum\limits_{i=1}^{N} \alpha_{i}(i)a_{ij}\right]b_{j}(o_{t+1}),~~ $$

(12)

$$\begin{array}{@{}rcl@{}} ~~1\leq t\leq T-1,~~~~1\leq j\leq N \end{array} $$

Termination:

$$ P(O|\lambda)=\sum\limits_{i=1}^{N} \alpha_{T}(i) $$

(13)

For each state s _j , α _j (t) stores the probability of arriving in that state having observed the observation sequence up until time t.

It is apparent that by caching α values, the forward algorithm reduces the complexity of calculations involved to N ² T rather than 2T N ^T. One may also define an analogous backwards algorithm which is the exact reverse of the forwards algorithm with the backwards variable: termination:

$$ \beta_{t}(i)=P(o_{t+1}o_{t+2}\ldots o_{T}|q_{t}=s_{i},\lambda) $$

(14)

as the probability of the partial observation sequence from t + 1 to T, starting in state s _i [34].

3 Experimental results and discussion

Conducting the tests connected with verification of a person based on information about facial asymmetry comes down to creating appropriate system. The system provides to add new users to database, to verify people already added to database and also to describe properly and code certain users. Depending on a teaching, recognizing model which is provided by hidden Markov models, it requires to form appropriate coded observation vectors. In the proposed method, the final observation vector consists of symbols, integers from 1 to 60. The length of feature vector is 15, which is a quite low value. For sure, such an assumption allows to learn the system quickly and add and verify new users. On the other hand, there are quite few analysed features in order to make proposed system accessible for big (above 500 people) user base. Further extension of number of asymmetric face features requires analysing larger face area and taking into consideration additional characteristic points located on a forehead, a chin, ears, and etc. However, additional features are dependent on haircut. The correct verification and identification for long hair may be difficult to fulfil. Only 15 features are taken into account in the process of testing the system.

All designated features are strictly dependent on the accuracy of the localization of face characteristic points. It is necessary to conduct initial tests before carrying out the proper tests connected with the identity verification. The initial tests are to compare the position of characteristic points designated on a face in automatic way with those made by hand. It is a time-consuming process, so the initial tests were executed on 250 different faces. Comparing results of point localization in automatic and manual way are presented in Table 1. Approximation is correct if the difference of the distance between automatically designated point is not bigger than 0.2 Muld from the manual one.

Table 1 Comparing results of localization of points in automatic and manual way

Based on the above table, one may conclude that the mouth corners are the easiest to locate. It is connected with the mouth colour which is specific and quite easy to distinguish. The worst to locate are the points defining the nose width at its widest point. Depending on the depth at the levels of luminance, it does not always allow the algorithm to work properly. In the case of eyes, there are several factors affecting on eye corner localization, so it is easier to receive correct result.

When the automatic localization of characteristic points is effective, then tests verifing the identity may be done. First, accept the strategy of creating an observation vector. The first 11 features would be described by the values of observation symbols from 1 to 40. Other four features would be in the range from 1 to 60, if the designated point of intersection would be within a code network. When the designated point is outside the code network, the observation vector would be cut back on the analysed feature. The tests were performed to verify the identity using hidden Markov models for the two variants. The first variant assumed that the length of observation vector is 15 features, with the possibility to reduce to 11. The second variant assumed much greater importance of the features resulting from localized points on code networks, depending on a five times duplication of each properly designated observation. The observation vector length for the second variant was contained in the range from 11 to a maximum of 31 features. All tests were performed on users’ basis containing 500 different faces for own database (MK DB) and CASIA database [39], taken in certain light conditions and with high resolution. For each user, two photos were considered, one to learn Markov models and the other one to test. In addition, the selected pictures were only of those users for whom all designated characteristic points were within the correct location (according to the criteria set out in the initial tests). The results of identity verification obtained for the first and second variant of the tests were given in Table 2.

Table 2 Results of tests for identity verification based on the asymmetry of the face, for the first and second variant of the test

The results indicate that more data in the observation vector allows to gain better verification efficiency, thus the second variant is better than the first one. It is not a big difference to demonstrate clearly the superiority of any of the proposals.

Used hidden Markov models decide about the advantage of tests’ second variant over the first one. This tool is able to generalize, even so with significant differences in defining observations 12,13,14 and 15. It is also possible to interpret correctly the asymmetric features, of course, with the enlarged vector. Please note that there were considered only a certain photos for the above-mentioned studies. The automatic localization of face characteristic points on these photos was included in the accepted criteria.

The described cases concerned a pretty selective tests. Only those images that met the basic criteria were analysed, namely, the correct localization of face characteristic points on the face. It is known that a lot of distinguishable features must be obtained—in order to identify or verify a person correctly. It concerns especially large databases. In order to illustrate the real possibilities of this method, the tests were conducted again. They were carried out in accordance with the assumptions of first and second variant but on randomly selected face images from available database. Also, in these tests, for each user, two different pictures were selected to learn Markov models and to test. In addition, also the photos with a lower resolution and lower quality were used compared to those used in the previous tests. The study was conducted in terms of false acceptance and false rejection rates, regardless of the proper functioning of the certain stages of the method. All the results from the first and the second variant tests are given in Table 3.

Table 3 Results of tests for identity verification based on the asymmetry of the face, for the first and second variant of the test, with no preliminary selection images

It can be expected that in conditions close up to the actual, error rates reach a worse value. The tests provided one very essential information. Namely, the creation of observations based on straight lines through characteristic points, intersecting in the face area, introduces too big discrepancy during encoding. This leads to worse functioning of the whole method, during the preliminary selection of images. Finally, it is worth mentioning that both the false acceptance rates, as well as the false rejection rates, were determined based on the accepted experimentally threshold value, which allowed the use of hidden Markov models.

The results shown in Tables 4 and 5 confirm how valuable the method is, see Fig. 12. In comparison to other methods, there were achieved much better results, even than methods using three-dimensional data [11, 40].

Table 4 Comparison to other face recognition methods base on asymmetry

Table 5 Comparison to other mouth detection methods

4 Conclusions

The face asymmetry is one of the newer trends in research related to biometrics. As every new trend, it needs a lot of tests and many new ideas in order to use it in a fully commercial and efficient system. The method described in this paper relies on pictures of colour faces, taken in a very good quality. It is known that colour pictures depend on lighting conditions rather than the greyscale pictures. However, at the present stage of research, it is hard to reject the information about the colour, due to simpler functioning of certain algorithms in RGB space.

The proposed method of asymmetry uses actually only six characteristic face points. Other elements of algorithms form a template of facial asymmetry on these six characteristic face points. It is clear that this is definitely not enough elements to create a global system for a large number of users. It should be focused on using the information about facial asymmetry from point of supporting identification and verification systems, based on other biometric features. For example, a system of person recognition based on the face (i.e. the look of the face) could be supported by asymmetry information. Facial asymmetry information is often used in medicine as an element indicating the degenerated changes or any disease [41]. In the work [26], the author mentions the use of facial asymmetry to identify physical and mental diseases based on the so-called ophthalmic geometry. However, as it turned out after the studies in this work, the information about the point of the two straight lines’ intersection cannot be used, when lines are designated based on points located very close to each other with regard to designated point of the intersection (Table 5). This approach causes that an error in determining the base point at the level of even one pixel would result in a large difference in the output (i.e. the intersection of two straight lines).

The use of hidden Markov models for the person verification based on facial asymmetry is possible if the right conditions are provided to learn and test the pictures. The models manage better if base points are determined properly, and the template of facial asymmetry has every time a similar shape (within a given user). In other way, the model may give false results, especially when there is a large database of pictures. However, if the errors resulting from the faulty determining of the face characteristic points are being corrected, it is worthwhile to use hidden Markov models, which were shown in the second variant of the tests with a preliminary picture selection. First of all, Markov models allow to analyse the data presented in the form of observation vector with different lengths, which increases the system flexibility, and in addition, it is easy to control the verification process based on the thresholding of a computed response value to the input data by the model.

In summary, the information about facial asymmetry may be used as a further feature in hybrid systems, especially if the number of users in the system database is quite significant. In the case of a small number of users (up to several dozen), the correct identity verification is possible at a very safe level.