Contactless hand biometrics for forensics: review and performance benchmark

Abstract

Contactless hand biometrics has emerged as an alternative to traditional biometric characteristics, e.g., fingerprint or face, as it possesses distinctive properties that are of interest in forensic investigations. As a result, several hand-based recognition techniques have been proposed with the aim of identifying both wanted criminals and missing victims. The great success of deep neural networks and their application in a variety of computer vision and pattern recognition tasks has led to hand-based algorithms achieving high identification performance on controlled images with few variations in, e.g., background context and hand gestures. This article provides a comprehensive review of the scientific literature focused on contactless hand biometrics together with an in-depth analysis of the identification performance of freely available deep learning-based hand recognition systems under various scenarios. Based on the performance benchmark, the relevant technical considerations and trade-offs of state-of-the-art methods are discussed, as well as further topics related to this research field.

1 Introduction

The introduction of fingerprint biometrics in the early twentieth century as valid person identification method has led to the solving of known crimes, e.g., the death of Francesca Rojas’ children at the hands of their mother in Argentina [1]. For suspect identification, fingerprints are usually collected at crime scenes. Afterwards, a recognition system may perform a one-to-many (1: N) comparison of the collected evidence against stored biometric references in order to confirm the guilt or innocence of a suspect [2]. With the rapid development of surveillance applications, other types of biometric characteristics, such as face and gait, have emerged as an alternative to fingerprints for forensic investigations [3]. These developments have, nevertheless, driven criminals to become smarter and to hide their visible biometric characteristics, e.g., their face and fingerprints. In this way, they avoid detection by respective recognition systems. An important aspect of forensic investigation is the identification of offenders and victims from evidence images. Identification from images of evidence is very problematic if no obvious characteristics such as face or tattoos are available. Due to the prevalence of smartphones and consumer cameras, evidence is increasingly available in the form of digital images taken in uncontrolled and uncooperative environments, e.g., images of child pornography and images of terrorists, where offenders often hide or cover their biometric characteristics, e.g., their face. However, their hands may be visible.

To assist law enforcement in identifying wanted criminals and missing victims, the use of contactless hand biometrics shows high potential, as hand images not only have less variability compared to other biometric modalities, but also exhibit strong and diverse characteristics that remain relatively stable after adulthood [4]. The hand is a primary biometric characteristic that provides distinctive features for biometric recognition. Besides fingerprint biometrics, former hand-based techniques analysed the features of a particular area of the hand, e.g., palm [5] and finger knuckles [6], and others focused on the analysis of the geometry of the hand [7] using handcrafted methods. Some of the more recently proposed schemes extracted traditional ridge and valley features, e.g., minutiae, from contactless 2D palm images [8] and other approaches analysed 3D properties of finger knuckles [9]. Hand geometry-based pipelines mostly focused on the silhouettes or morphology of the hand for a fixed pose [7, 10]. Finger and palm veins have also gained a lot of attention, especially in commercial systems, as they enable contactless sensing and are more resistant against forgeries (i.e. spoofing, presentation attacks) as the vessels are only visible in infrared light [11].

Deep neural networks (DNNs) have successfully replaced traditional approaches like handcrafted techniques with powerful architectures that can learn more robust features from full hand images in a feed-forward manner. In 2019, Afifi [12] proposed a public database of contactless hand images together with a convolutional neural network (CNN) method for gender classification and subject identification. Building upon this method, more recent CNN-based algorithms have exploited attention mechanisms [4, 13] or vision-transformers [14] to improve the baseline identification performance reported in [12]. These schemes were mainly evaluated in closed-set identification scenarios where the searched identity is known to be included in the database of enrolled references. In addition, the databases used in said studies contain controlled hand images with few variations in background context, hand pose and finger gestures—properties that often have high variations in images processed during forensic investigations.

Hand images collected in forensic scenarios exhibit challenging properties, such as high variations in background context, hand pose and finger gestures. Motivated by the fact, in this work, we conducted a comprehensive study on the identification performance of several current deep learning-based hand recognition approaches in various scenarios. The main contributions of this article are:

• A comprehensive literature review on hand-based recognition systems which provides a taxonomy that conceptually categorises hand recognition methods for biometric identification. In contrast to other scientific reviews that analyse a specific area of the hand, e.g., palmprint [8], we provide a general overview of all hand-based approaches with a particular focus on full hand-based methods which are of special interest for forensic investigations.

• A performance benchmark of the freely available full hand-based approaches under several scenarios. We selected only deep learning-based techniques, as they have shown the best identification performance in most recognition tasks, e.g., face recognition [15]. Other methods, such as those based on hand geometry [10], are not taken into account in our analysis, as they are sensitive to images taken in uncontrolled environments, resulting in low identification performance. For the evaluation, we consider easy or difficult scenarios which are encountered in forensic investigations. Conducted experiments are compliant with the metrics defined in the international ISO/IEC 19795-1 for biometric performance testing and reporting [16].

• The evaluation of the impact of tattoos on hand recognition. In this work, we evaluate how the use of tattoos in the dorsal area of the hand affects the identification performance of state-of-the-art hand recognition systems.

• A discussion on technical considerations and trade-offs that are relevant from a forensic investigation perspective as well as further topics relevant to this research field.

Contrary to our previous conference article [17], this work provides a comprehensive review of the literature on hand-based recognition systems, extends the performance evaluation of these approaches to several practical scenarios that are of interest in forensic investigations, and sets up a discussion on technical considerations and trade-offs that are also relevant from a forensic investigation perspective.

The remainder of this article is organised as follows: Sect. 2 provides an overview of background information and outlines current hand recognition systems. In Sect. 3, technical characteristics of open-source hand detection and recognition systems are outlined, together with the properties of currently available databases. An in-depth performance benchmark reporting the identification performance of freely available hand-recognition schemes under different scenarios is presented in Sect. 4. In Sect 5, we present the discussion of the experimental results reported in the performance benchmark along with further topics in this research field. The conclusions are finally drawn in Sect. 6.

2 Literature review

The anatomy of the hand is the key to determine the different categories of hand-based biometrics. In general, a hand consists of a broad palm with five fingers, each attached to the joint called the wrist. The back of the hand is formally called the dorsum or dorsal of the hand [18]. In this section, the current state-of-the-art is presented. First, we introduce and describe the taxonomy summarising the current techniques that analyse different anatomical areas of the hand for subject recognition. This is followed by a comprehensive survey of existing methods focusing on the proposed taxonomy.

Fig. 1 shows a proposed taxonomy that categorises existing methods for subject recognition which focus on hand biometrics. For this purpose, hand-based techniques can be separated into two main approaches: hand part-based (Sect. 2.1) and full hand-based (Sect. 2.2) techniques. While the algorithms in the first category analyse specific areas of the hand, such as knuckles, fingerprints, palmprints, and veins, those in the second category focus on the full frontal or dorsal part of the hand. Note that some of the described approaches are based on hybrid schemes, combining, e.g., the analysis of finger knuckle and palmprint.

Fig. 1

Taxonomy of methods that analyse partial or full hands for subject recognition. As illustrations, the different parts of the hand on which the taxonomy is based are shown in the corners

2.1 Hand part-based

Over the years, several studies have demonstrated the individuality of the different areas of the hand for the purpose of recognising persons [8, 19, 20]. The key idea behind the hand part-based approaches is the analysis of single or multiple areas of the hand, which contain discriminative information for the recognition of individuals.

The discovery of fingerprints in the nineteenth century and their acceptance as a characteristic for person recognition at the beginning of the twentieth century has led to the development of numerous scientific investigations [21]. Recently, health concerns related to the rise of the SARS-CoV-2 coronavirus have led to the analysis of features that can be extracted from contactless fingerprints, palmprints and finger knuckles, i.e. these types of biometric characteristics can be acquired using, e.g., a smartphone camera, without any need to touch a sensor surface. Traditional global (e.g., orientation field, ridge density, or fingerprint types) and local (e.g., minutiae and pores) features can be extracted from fingerprint images; some of them also correspond to other types of biometric characteristics, such as vascular [22] and palm [8] patterns. While some authors analysed local characteristics, such as minutiae together with the principle lines and wrinkles of the palmprint ridge pattern [8], others have directly fed DNNs with RGB images of the palm [23,24,25] and finger knuckle images [26, 27]. Some articles also proposed multi-type algorithms based on the fusion, e.g., between the palmprint and knuckles [28] or the knuckles together with the fingernails [29]. In addition, the 3D technology to extract more discriminative features from the fingerprint [30, 31], palmprint [32], and finger knuckles [9, 33,34,35,36] have been employed. The interested readers are referred to, e.g., Alausa et al. [8] for a comprehensive review about techniques used for palmprint recognition, Cheng and Kumar [36] for an overview on finger knuckle-based schemes, and Prietniz et al. [37] and Chowdhury and Imtiaz [38] for scientific advances on contactless fingerprint recognition. Scientific studies on finger and palm vein recognition were surveyed by Shaheed et al. [39] and Wu et al. [22].

Despite the advances achieved by the previously mentioned approaches, most techniques have not yet been evaluated on uncontrolled databases that are closer to real-life scenarios. That is, data containing various external factors, such as environmental conditions and hand gestures to hide, e.g., the finger knuckles or the palm area, can affect the recognition performance. In addition, the recent 3D techniques rely on the integration of special sensors (e.g., depth sensor) within the capturing devices, which limits their usability in forensic investigations. The near-infrared (NIR) sensors are also a step forward in the development of high-performance recognition systems [22, 39]. However, the integration of these NIR sensors also hinders the applicability of associated detection techniques in forensic investigations.

2.2 Full hand-based

As the focus of this article is the analysis of the feasibility of full hand-based recognition techniques for forensic investigations, we will describe the different approaches (i.e. Hand geometry—Sect. 2.2.1 as well as Learned features—Sect. 2.2.2) that have commonly been used for this purpose. The main characteristics of the techniques outlined in this section are also summarised in Table 1.

Table 1 Performance overview of full hand-based biometric recognition approaches

2.2.1 Hand geometry

The anatomy of the hand shape depends on its geometry, its length, the finger’s width and the span of the hand in different dimensions. Sharma et al. [40] combined features extracted from finger peaks and valleys and distances between hand landmarks for biometric verification. To yield an equal error rate (EER) of 0.52% over a controlled database of 230 subjects [5], several pre-possessing steps that included segmentation, rotation, and alignment of the hand were performed. In order to improve the biometric performance of geometry-based recognition systems, some studies fused hand geometry with other types of biometric characteristics, such as inner knuckle prints and the palmprint. Anitha et al. [41] proposed a biometric recognition system by combining hand geometry features with inner knuckle prints. Following the pre-processing steps utilised in [40], the authors detected the edges of the hand and rotated it accordingly so that the reference points of the assessed hands were aligned. Chen and Wang [7] finally combined the geometric features of the hand with geometric characteristics extracted from the palmprint. For more recent works covering the topic of hand geometry-based recognition, the interested readers are referred to [44].

In general, biometric systems based on hand geometry are widely deployed in access control applications because they are easy to use, and have high public acceptance and good integration capability [45]. However, the geometry features of the hand are not considered suitable for use in a large-scale personal identification of individuals, as the sole geometric properties of the hand are not very distinctive [18].

2.2.2 Learned features

In order to improve the balance between applicability, user convenience, and recognition performance of previous approaches, the latest techniques map whole hand images acquired in the visible spectrum into a latent representation using DNNs. Afifi [12] introduced an annotation-rich hand database (referred to in the scientific literature as 11K Hands) consisting of 11,076 high-quality hand images captured in the visible spectrum. In addition, the same author proposed a dual-stream CNN-based algorithm whose recognition performance values (i.e. correct identification rates (CIRs) ranging from 94% to 97% in Rank-1 for the palmar and dorsal area, respectively) provided a starting benchmark for future investigations. Following the above idea, Baisa et al. [13] recently proposed a dual-stream CNN approach which learns both global and local features of the hand image. The experimental results reported a CIR in Rank-1 of around 95% on 11K Hands [12]. Baisa et al. [4] extended this architecture by including an extra stream and incorporating both channel and spatial attention modules in branches. An improvement in recognition of around 3% (i.e. CIR = 98.05%) was achieved on the right palm images compared to the 95.83% obtained in [13]. In the same study, other CNNs were evaluated for the hand recognition task, e.g., ABD-Net [42] and RGA-Net [43], resulting in similar recognition performance to the one in [13]. Finally, Ebrahimian et al. [14] evaluated the feasibility of using Vision Transformers for hand recognition, resulting in a CIR of 99.4% on a small subset of 11K Hands consisting of 30% of the images. In spite of the results achieved for the techniques described in this category, a proper evaluation remains missing that includes more realistic and challenging hand images (e.g., images in NTU-PI-v1 [23] or HaGRID [46]) and scenarios (e.g., open-set scenario). The above methods were only evaluated in a closed-set scenario over controlled images stemming from the datasets: 11K Hands [12], Hong Kong Polytechnic University Hand Dorsal (HD) [47] and IIT Delhi Touchless Palmprint Database [5] (see Sect. 3.1 for details).

Motivated by the fact that previous scientific studies have shown that there is a reliable balance between applicability (i.e. they can be used in most applications, including forensic investigations), user convenience (i.e. hand images are acquired with a contactless capture device without requiring any subject effort) and recognition performance (i.e. the results summarised above demonstrated that the palmar and dorsal areas contain distinctive information for subject recognition), we conducted a comprehensive benchmark of the performance of the available schemes on uncontrolled images covering realistic and challenging scenarios that are of interest in forensic investigations (see Sect. 4).

3 Resources

To facilitate academic and industrial research on hand recognition, an overview of available databases (Sect. 3.1) is provided. Technical characteristics of open-source hand recognition systems are also outlined (Sect. 3.2). Since hand detection is an important step in the recognition pipeline, open-source techniques employed for hand detection are also presented (Sect. 3.3).

3.1 Databases

Table 2 summarises the main characteristics of the available databases for hand recognition. Note that most of the databases consist of several samples (more than 1,000 instances) from a large number of subjects ranging from 114 to 43,669. Samples in most of the databases were acquired with high-resolution capture devices and in controlled scenarios, i.e. controlled background, fixed gesture and high image quality, which are unrealistic properties in the context of forensic investigations (see Fig. 2a–e). In contrast, images in NTU-PI-v1 [23] and HaGRID [46] were captured under uncontrolled parameters, i.e. varying the aforementioned attributes (see Fig. 2f-g). The two databases 11K Hands [12] and NTU-PI-v1 [23] also include information on demographic attributes of the subjects that could potentially benefit research related to both demographic fairness in full hand-based recognition systems and forensic analysis. For example, age and gender attributes might be of interest, e.g., to reduce the false positive rates in biometric identification systems. Hand gesture variations in HaGRID images are also useful for the development of automatic hand gesture detection techniques.

Table 2 Summary of available databases for hand recognition

Fig. 2

Examples of images in controlled (a)-(e) and uncontrolled (f)-(g) databases

3.2 End-to-end full hand recognition systems

Table 3 summarises technical details of open-source hand recognition systems which might be of interest to industry and academia. Note that the works with hand recognition systems implemented in Matlab provide the trained weights, while the studies implemented in PyTorch only describe the training and testing pipelines on the respective websites. From a business point of view, one approach (i.e. MBA-Net [4]) does not provide licensing information and can therefore be used in commercial applications. The remaining works comply with the MIT license which permits unrestricted use of the Software, i.e. without limitation on the rights to use, copy, modify, merge, publish, distribute, sublicense and/or sell copies of the software. Finally, it should be perceived that MBA-Net reports the best balance between recognition performance (i.e. it yields one of the best CIR values for the closed-set scenario) and spatial efficiency (i.e. it is the lightest architecture with a model size of 173 MB). To benchmark the hand recognition performance, we selected the most competitive approaches, i.e. MBA-Net [4] ABD-Net [42], and RGA-Net [43].

Table 3 Summary of the technical characteristics of open-source hand recognition systems

3.3 Hand-based detection systems

Table 4 summarises the technical details of the state-of-the-art approaches employed for hand detection. Note that all studies make the trained weights available and provide licensing information. In contrast to all works described in Table 3, three of the described hand detection-based algorithms can only be used for commercial purposes, namely Hand-CNN [50], ContactHands [51], and BodyHands [54]. In contrast, Google’s MediaPipe solution [52] and HandLer [53] comply with the Apache-2.0 license that limits, e.g., its use in market applications. The former also proposed a lightweight CNN architecture whose weights are 4 MB in size. This characteristic, together with its usability and performance, has led to Google’s MediaPipe solution being widely used in a high number of academic research projects. Note that hand detection is outside the scope of this study, as the hand images in the evaluation databases were previously cropped by the authors [12, 23, 46]. As a future research topic, we recommend evaluating the influence of hand detection on the performance of contactless hands recognition.

Table 4 Summary of technical characteristics of open-source hand detection approaches

4 Performance benchmark

At present, available full hand-based approaches depicted in Table 3 have been mostly evaluated on non-challenging controlled scenarios (e.g., closed-set on 11K Hands). Therefore, a proper evaluation for more realistic and difficult settings, e.g., open-set and cross-database identification on uncontrolled databases, remains missing. To provide insights on the performance of the state-of-the-art full hand-based approaches, a set of experiments ranging from easier to more challenging scenarios is outlined in Sect. 4.1. The results for each experimental setup are then presented in Sect. 4.2.

4.1 Experimental setup

Table 5 summarises the experimental protocols to evaluate the performance of available full hand-based algorithms (see Table 3) from easier to challenging scenarios. For this purpose, the most recent databases in Table 2 are selected, i.e. 11K Hands, HaGRID, and NTU-PI-v1. We then defined six configurations for evaluation: easy controlled (Table 5, row 1), easy uncontrolled (Table 5, row 2), challenging uncontrolled (Table 5, row 3) databases, and cross-database on challenging databases (Table 5, rows 4–6). Note that most experimental protocols focus on the analysis of the palm, as this is the visible area of the hand in difficult database images. Since high performance degradation is expected for protocols 2–6, the open-set scenario is only evaluated in configuration 1. In the open-set evaluation, fivefold validation sets on the 11K Hands subjects were randomly created. In all experiments, subjects used for scheme training are different from those used for enrolment and transactions. To avoid bias, subjects and their respective images were randomly chosen for the enrolment as well as the and reported in compliance with the metrics defined in the international ISO/IEC [16]:

• (i) Identification rate (IR), which is presented as a graph of the cumulative matching characteristic (CMC) at different Rank values.

• (ii) False negative identification rate (FNIR), which is defined as the proportion of a specified set of identification transactions of subjects registered in the system for which the correct subject reference identifier is not among those returned.

• (iii) False positive identification rate (FPIR), which is defined as the proportion of non-enrolled subject identification transactions for which a reference identifier is returned.

Table 5 Experimental setup characteristics

4.1.1 Experimental databases

As mentioned in Sect. 4.1, the most recent databases were selected, i.e. 11K Hands, HaGRID, and NTU-PI-v1, where the latter reflects the diversity of hands in the scenarios, where there is no control over image acquisition parameters or subject cooperation and images are taken without any intention to recognise the hand. Diversity is represented by significant differences in hand gestures, point of view, lighting, background, image quality and resolution [23]. Since HaGRID [46] was initially proposed for gesture recognition purposes, it contains many different gestures. We only selected those similar to 11K Hands and NTU-PI-v1 from its test set, i.e. palm and stop gesture. In the experiments, those hand images having extremely low quality were filtered out by using a Laplacian variance approach [55]. A low Laplacian variance indicates an edge absence and therefore a blurred image. Fig. 3 shows the image quality distributions computed on HaGRID and NTU-PI-v1 as well as the selected threshold (λ = 10) to remove the low-quality images. λ was selected by a visual inspection of the images. Table 6 provides details of the databases used in the performance benchmark after removal of the low-quality images and single-sample subjects. In case of the 7^th protocol (last row), 10 tattooed versions of each hand image are generated from the respective evaluation subset using the proposed method described in [17], resulting in 10,420 generated images for right dorsal (see Fig. 4, second row). It is worth noting that not all images used as biometric transactions in the dorsal evaluation were processed, due to a failure in the detection of their landmarks. In the final evaluation of the tattoos, 33 out of 71 identities are considered for the right dorsal.

Fig. 3

Image quality distributions for HaGRID and NTU-PI-v1 and the respective filtered threshold (black dashed line)

Table 6 Characteristics of databases used in the experimental evaluation

Fig. 4

Examples of 11K Hands images (first row) and their respective tattooed hands (second row)

4.1.2 Implementation details

All algorithms used in the performance benchmark were implemented in PyTorch [56] and trained utilising a Nvidia A100 Tensor Core GPU with 40GB of GPU Memory. For the training and testing of the systems, we took the parameters as indicated in their corresponding articles. The image size was set to 256 256 pixels for ABD-Net [42] and RGA-Net [43], and 356 356 for MBA-Net [4]. The networks were initialised with their pre-trained weights in ImageNet [57] and trained for 70 and 100 epochs for the experiments with controlled and uncontrolled data, respectively, using the Adam optimiser with a learning rate of 0.02. As indicated in [43], the RGA-Net architecture was trained for 600 epochs in all cases. More technical details on the network architectures can be found in the papers MBA-Net [4], ABD-Net [42], and RGA-Net [43]

4.2 Experimental results

This section summarises detailed results computed for the experimental protocols, defined in Table 5. To make the result’s discussion more accessible to the readers, the next sections are named with respect to the Experiment name in Table 5, except for the cross-database experiments which are reported in Sect. 4.2.4.

4.2.1 Easy controlled

The Easy Controlled scenario aims at evaluating the most competitive full hand-based approaches under ideal conditions. For this purpose, the 11K Hands database was selected, containing images of hands acquired with few gestural variations and static white backgrounds. Fig. 5 depicts the CMC curves for dorsal and palmar images in 11K Hands in a closed-set scenario where the same subjects participate in both enrolment and biometric transactions. Note that, MBA-Net yields the best identification performance for dorsal and palm images under controlled conditions: IRs greater than 96.7% at the Rank-1 are computed for both hand regions. This algorithm also achieves an IR of 99.9% in the Rank-5, indicating that the biometric identifiers of transactions are retrieved by the system with almost 100% success in the top 5 positions of the candidate list. From a forensic point of view, high IRs for Ranks above 1 are still interesting, as additional candidates might be searched suspects. Regarding ABD-Net and RGA-Net performances, degradation of their IRs with respect to the ones attained by MBA-Net can be observed: IRs ranging from 84% to 92% are yielded for both hand regions. Both schemes are also of interest for use in forensic investigations, due to the reported IRs for ranges above 1, i.e. IRs 94.8% for ABD-Net and IRs 91.2%, respectively, at the Rank-5.

Fig. 5

CMC curves for the controlled dorsal (5a) and palm (5b) images in 11K Hands

The identification performance of the available full hand-based methods is also reported in Table 7 for an open-set scenario in which some biometric identifiers remain missing from the enrolment. Similar to the results in Fig. 5, MBA-Net achieves the best performance, resulting in a FNIR = 10.18% for a high-security threshold, i.e. FPIR = 0.1%—1 out of 1000 non-mated transactions is accepted, while at most 10 out of 100 mated transactions are rejected by the system.

Table 7 Identification performance (%) for palm images in 11K Hands

In contrast to the results illustrated in Fig. 5b, the identification performance of RGB-Net outperforms the one yielded by ABD-Net for high-security thresholds, i.e. FNIR ≥ 11.88% for RGA-Net vs. FNIR ≥ 30.48% for ABD-Net at a FPIR ≤ 20%. These results state that RGB-Net is more robust to unknown subjects that are not enrolled in the system, compared to ABD-Net, which is a desired behaviour for forensic investigations.

4.2.2 Easy uncontrolled

This set of experiments evaluates the identification performance of full hand-based systems on an uncontrolled database in which the background context looks different from that represented by 11K Hands images (see Fig. 2). For this purpose, we computed the performance of the schemes in the HaGRID database and depicted it in Fig. 6.

Fig. 6

CMC curves for the easy uncontrolled HaGRID database

Note that, the algorithm’s performance considerably dropped with respect to the ones reported in Fig. 5b. In particular, the best-performing architecture for the controlled database (MBA-Net) achieves an IR of 54.10% at Rank-1 which is almost half of the result outlined in Fig. 5b for the same Rank (i.e. 96.80%). Subsequently, RGA-Net and ABD-Net also have a decrease in accuracy down to 30.60% and 59.80%, indicating the need to develop and adopt hand segmentation strategies as a pre-processing step. Despite the performance deterioration, we can observe that both MBA-Net and ABD-Net yield IRs ≥ 80% for Rank values ≥ 10 that are still of interest in forensic investigations.

4.2.3 Challenging uncontrolled

We then selected the NTU-PI-v1 database to evaluate the performance of the systems on difficult images with varying background context, finger gestures and hand poses. Fig. 7 shows the algorithm’s performance plotted for different Rank values. Note that, the identification performance of the evaluated schemes suffers a significant deterioration, which is even worse than the results shown in Sect. 4.2.1. IR values computed for all networks are lower than 40% for Ranks ≤ 20. In contrast to the previous experiments, ABD-Net outperforms MBA-Net for high-Rank values (i.e. Rank ≥ 6), indicating that the former also analyses non-textural features and is therefore more robust against uncontrolled databases. Since the images of NTU-PI-v1 show high variations in hand poses, we strongly believe that an alignment of the hands could lead to improved results compared to those in Fig. 7.

Fig. 7

CMC curves for the challenging uncontrolled NTU-PI-v1 database

4.2.4 Cross-database

In the last set of experiments, we evaluate the generalisation capabilities of the embedding representation computed by the systems. For this purpose, we focused on the three last train–test configurations defined in Table 5 (rows 5–7) and computed their respective CMC curves; compare Fig. 8. As expected, a significant performance deterioration can be seen, with respect to the above intra-database evaluation. Note that, on the one hand, training the systems on controlled images (i.e. 11K Hands) with few background variations leads to the highest performance deterioration on uncontrolled images like those from HaGRID (down to 34.90% for ABD-Net in Fig. 8a), and on the other hand, the inclusion of images with varying background context and finger gestures improves the algorithm’s performance shown in Fig. 8a for high-Rank values (Rank ≥ 5 in Fig. 8b). However, this is not sufficient to achieve the results outlined in Fig. 6, having an IR of 76.50% for the same Rank values.

Fig. 8

Cross-database performance evaluation representing the last three train–test experimental configurations in Table 5

In contrast to the above results, an improvement in identification performance was achieved for images in NTU-PI-v1 (Fig. 8c), when systems are trained with data from the HaGRID database. While MBA-Net reports IRs ≥ 30.10% for Ranks ≥ 5 using HaGRID as the training database, this yields IRs ≥ 22.70% for the same Rank ranges within its intra-database evaluation (see Fig. 7). These results are mainly due to the fact that images of hands with few variations in finger gestures (e.g., HaGRID images) introduce less bias (i.e. high generalisation) in CNN training than those with large gesture variations (e.g., NTU-PI-v1 samples).

4.2.5 Impact of tattooed hands on recognition

The use of tattoos on the hand has recently gained popularity. In this section, the results of the impact of the use of tattooed hands on the recognition performance of the systems evaluated for closed-set and open-set scenarios, respectively, are presented.

4.2.5.1 Closed-set evaluation

In the experiments we evaluate the impact of tattooed hands on the recognition performance of the available systems. Fig. 9 shows the identification rates for non-tattooed (9a), tattooed only on the probe (9b), and tattooed on both reference and probe (9c) dorsal hands for the closed-set scenario. To compute IRs at different rank values, we split the database into 10 disjoint sets of enrolment and biometric transactions, each time randomly selecting one sample per subject for enrolment and the remaining samples for identification transactions. Then the mean and standard deviation (std) are reported. For biometric transactions of tattooed hand images, we enrolled either a non-tattooed (9a) or tattooed (9c) reference from the same probe subject. To simulate a real scenario, reference and probe hand images were generated using the same tattoo template in the latter case (9c).

Fig. 9

CMC curves reported by the evaluated systems on non-tattooed (a), tattooed only on probe (b), and tattooed on reference and probe (c) images from 11 K Hands

Comparing the results in Fig. 9a, all networks report on average a performance deterioration for tattooed hands: the IRs for the best-performing approaches (i.e. MBA-Net and ABD-Net) decrease down to 97% in Rank-1. Furthermore, the std values increase regarding the ones depicted for non-tattooed hands. This deterioration in recognition performance is due to the fact that the features calculated by both architectures describe mainly textural details. Therefore, they are prone to fail on tattooed hands. In contrast to MBA-Net and ABD-Net, RGA-Net obtains on average similar results for tattooed and non-tattooed hands, i.e. IRs in around 87%. However, compared to the other methods, this technique obtains the worst std values for subjects with tattooed hands.

Note that the biometric performance yielded by the networks when both reference and biometric transactions contain tattooed hands (see 9c) is similar to that of non-tattooed hands in Fig. 9a. A direct result of this observation is focused on the use of images of tattooed hands to train the algorithms. Thus, the performance shown in Fig. 9b might be significantly improved.

4.2.5.2 Open-set evaluation

The identification performance of the available hand-based methods is also reported in Fig. 10 for an open-set scenario. To compute mated and non-mated comparisons, we perform a tenfold cross-validation evaluation. Thus, each time, the subjects belonging to the validation fold at hand are employed for computing the non-mated comparisons, while the remaining subjects from the other subsets are used for the mated comparisons. For the assessment of the impact of the tattoos, the non-tattooed subjects in the validation fold in question are replaced by the same subjects with tattooed hands.

Fig. 10

DET curves for left and right dorsal images

Similar to the results in Fig. 9, MBA-Net achieves the best performance (dark blue thick lines), resulting in a FNIR = 12.03% for a high-security threshold, i.e. FPIR = 0.1% on the dorsal images, respectively: 1 out of 1000 non-mated transactions is accepted, while at most 12 out of 100 mated transactions are rejected by the recognition system. Note that the use of tattooed hands significantly affects the performance of the architectures: FNIR values at a FPIR=0.1% are above 60% for the evaluated images, indicating the sensitivity of the current hand recognition systems to tattooed hands. Finally, we note that RGA-Net is less sensible to tattooed hands than the other approaches. This is due to some attention mechanisms which leverage both texture and shape properties.

4.2.6 Discussion

The naïve identification mode relies on a one-to-many (1:N) template comparison between the probe image and N biometric references belonging to N different enrolled subjects. This operational method is commonly used in forensic investigations to retrieve, e.g., the biometric identifier of a wanted criminal or to find a match within a list of missing individuals. The comprehensive study conducted in this manuscript through several experimental protocols showed that current full hand-based approaches performed well on controlled hand images having few finger gestures and background context variations (see Table 8). IR values in around 99.90% at Ranks $\ge$ 3 on palm and dorsal hand images from 11K Hands database show the networks suitability for many real applications, such as hand authentication.

Table 8 Summary of identification rates (%) in Rank-1 achieved by hand recognition systems in several scenarios

In the Sects. 4.2.2 and 4.2.3, we observed that variables such as the finger gestures and variations in the background context caused a significant deterioration in the performance of the algorithms, resulting in undesirable accuracy for forensic investigations: IRs decreased down to 12.50% at Rank-1 for challenging NTU-PI-v1 images. We noticed that HaGRID and NTU-PI-v1 images vary from pose to pose, therefore we strongly believe that a pre-processing step for pose alignment would improve the performance of the techniques. An additional segmentation step would also benefit those schemes.

In our exhaustive study, we only selected uncontrolled HaGRID images having the hand gestures palm and stop to evaluate the approaches. The results presented in Fig. 6 show a performance decrease for these uncontrolled images compared to the controlled hand images in 11K Hands. Therefore, we do confirm that the inclusion of more difficult hand gestures can significantly decrease the identification performance of the algorithms. Research into new techniques to disentangle these hand gestures and then transform them into controlled images of open hands in which the dorsal or palmar area is visible would be one solution to achieve high identification performance (see Fig. 11). In this context, 3D hand recognition, which, to our best knowledge, has not been addressed in the scientific literature, should be also investigated. To improve the IRs presented in Table 8, combinations of systems can also be considered. A fusion of the evaluated schemes at score or rank level is expected to enhance these results, in particular for more realistic and challenging scenarios, e.g., Challenging Uncontrolled and Cross-database Challenging, related to forensic investigations.

Fig. 11

Conceptual overview of the transformation of different hand gestures into the desired open hand, while preserving the subject identity

5 Further research topics

Based on the above study and results, we summarise further topics and recommendations in contactless hand biometrics in the section, e.g., soft biometrics (Sect. 5.1), template protection (Sect. 5.2), presentation attack detection (Sect. 5.3), workload reduction (Sect. 5.4), demographic fairness (Sect. 5.5), sample quality (Sect. 5.6), image synthesis (Sect. 5.7), and information fusion (Sect. 5.8) which may be of interest to industry, academia or forensics, and which should be used depending on the context of the application.

5.1 Soft biometrics

Soft biometrics are usually descriptive and have a semantic representation. They can be computationally inexpensive, discernible from a distance in a crowded environment, and require less or no cooperation from the observed subjects [19]. Extracting gender information from the hand can be traced back to forensic medicine and archaeology, which established gender from the hand morphology [19]. As a result of these studies, several approaches have been proposed to extract gender information from the hand. A comprehensive survey summarising gender classification-based techniques is outlined in [12, 19, 58]. Recently, other scientific works have also used hand information to estimate the age of subjects [59], resulting in a reliable classification accuracy of 96.50%. The hand-based age classification is a new field of research that needs to be analysed in depth for different demographic attributes of the subjects, e.g., work occupation. In general, soft biometrics can be used to improve the identification performance shown in Sect. 4.2. Reducing the candidate list given the gender information of the probe leads to a considerable reduction in both the FPIR and the system workload [60].

5.2 Template protection

Privacy regulations, e.g., the European Union (EU) General Data Protection Regulation 2016/679 (GDPR) [61], usually define biometric information as sensitive data. Unprotected storage of biometric references could lead to different privacy threats, such as identity theft, linking through databases or limited renewability [62]. In the context of biometric template protection, the majority of the scientific literature is focused on traditional types of biometric characteristics, such as fingerprint, palmprint, iris, face, voice, and vascular veins [63, 64]. To the best of our knowledge, few studies have addressed template protection in the context of hand recognition [65]. Since demographic information, such as age and gender, can be obtained from the hand, the application of template protection techniques to protect hand features should also be addressed in the future.

5.3 Presentation attack detection

Presentation attack detection (PAD) refers to the task of determining whether an input sample stems from a live subject or from an artificial replica. PAD is one of the most active biometric fields of research. Similar to biometric template protection, PAD techniques have been mainly proposed for a single type of biometric characteristic, e.g., fingerprint [66, 67], face [68,69,70], iris [71], voice [72, 73] and their combination in a multi-type PAD method [74]. With the increase in respiratory infections due to the SARS-CoV-2 coronavirus, contactless biometrics has experienced a broad development. The contactless fingerprint biometrics has received more attention to be used in real applications. It is therefore expected that other types of contactless biometric characteristics, such as the hand, can be used in the near future. In this context, hand-based attack presentation detection could benefit from the development achieved in other biometric characteristics mentioned above and lead to new PAD schemes for detecting attack presentations.

5.4 Workload reduction

In forensic investigations, researchers must efficiently handle a lot of data generated on a daily basis for the identification of criminals and victims. According to the study in [60], increasing the number of subjects enrolled in an identification system steadily raises, on the one hand, the response time of the system and, on the other hand, the false positive acceptance rates. Daugman [75] investigated the relationship between the number of subjects enrolled (i.e. N) and false positive acceptance rates, showing that the probability of false positives from one-to-many template comparisons increases rapidly to unacceptable levels with N. Current hand-based recognition schemes have the above limitation, as they perform a one-to-many comparison of the probe template against N enrolled subjects. It is therefore of particular interest for forensic investigations to develop workload reduction techniques, e.g., pre-filtering or binning based on soft biometrics, that can reduce both system response time and false positive acceptance rates.

5.5 Demographic fairness

The successful use of artificial intelligence to support humans in making complex decisions has led to the development of automated systems that, in many cases, are already outperforming and thus replacing humans. Recently, a number of ethical and legal issues have been raised, in particular in relation to the transparency, accountability, explainability and fairness of those systems [76]. Regarding the latter, an algorithm is considered to be biased (or unfair) if significant differences are observed in its performance for subjects from different demographic groups (e.g., women or dark-skinned people), thus benefiting certain groups of individuals [76]. In this context, biometric systems are not exempt from biased decision-making [77]. As hand recognition systems are mainly based on deep neural networks and can be deployed in several real applications, the need to reduce the negative effect of demographic fairness should be also addressed with new scientific studies in the near future.

5.6 Image quality

The performance of biometric systems mostly depends on the quality of the acquired data, which is influenced by numerous external factors, e.g., lighting conditions. Automatically evaluating the quality of data in terms of biometric utility can thus be useful to detect low-quality samples and then make decisions accordingly. According to ISO/IEC 29794-1 [78], the utility-based quality depends on the character and fidelity of a sample. Whereas the latter refers to the degree of similarity (e.g., of a blurred image) to its biometric feature of origin, the former is related to some uncontrolled biometric attributes (e.g., skin texture, scars). In our study, we used hand fidelity, computed using the Laplacian approach [55], as an image quality metric. Thus, blurred images of the hand were eliminated from the experimental evaluation. Despite the results obtained, our study lacks a proper evaluation of the performance of the schemes in terms of other utility-based quality metrics, e.g., image quality assessment (IQA) in [79]. Since the character attributes of the hand vary slightly from sample to sample, hand-based recognition systems can also exploit these properties to reduce the final candidate list by following, e.g., intelligent search based on image quality proposed in [80].

5.7 Image synthesis

Due to some privacy issues in the acquisition of biometric samples, new technologies such as Generative Adversarial Networks (GAN) and the recent Latent Diffusion Models (LDM) have proven their advantages in the generation of synthetic images that can replace real samples in many computer vision and pattern recognition tasks. Those generative models have been mostly employed in the synthesis of realistic facial images [81, 82]. In contrast to facial images, few works [38, 83] have explored hand image synthesis to alleviate the lack of databases needed to properly train and evaluate hand recognition systems. From a forensic point of view, there is a need for generative approaches that can create identity-preserving hand images that can be used to train robust hand identification systems. Such synthetic hand images can be generated under realistic environmental conditions by varying hand gestures and image quality.

5.8 Information fusion

The use of complementary information from different sources has mostly reported a performance improvement in several biometric tasks, e.g., subject identification [84] and PAD [66, 69, 73]. To improve the above identification performance shown in Sect. 4.2, investigators can proceed with strategies similar to those proposed in [85] in which hand recognition models were fused at different levels. Thus, the candidate list can be shortened and the FPIR values can be further reduced. In addition, forensic researchers can exploit latent representations of other soft biometric data, such as tattoos [86], to complement the hand representation and thus improve the identification of both criminals and missing persons. Tattoos, unlike other soft biometrics such as gender, age or race, contain more discriminative information to support the identification of individuals and are a useful indicator to track members of a criminal gang or organisation [86].

6 Conclusions

The use of hand biometrics has emerged as a potential candidate to assist law enforcement in identifying both criminals and victims. In this study, we summarised different hand-based techniques that can be classified into different categories. Based on which part of the hand current recognition systems analyse, we defined a taxonomy with two main categories: hand part-based and full hand-based schemes. This was followed by a comprehensive review outlining the advantages and disadvantages of the algorithms. Afterwards, an in-depth performance evaluation of available full hand-based pipelines led to several issues and open challenges that need to be addressed in future directions. In particular, improving the identification performance of the current system (i.e. a low IR at Rank-1 of around 12% for challenging scenarios) through, e.g., the use of soft biometrics as a workload reduction approach is of utmost importance for forensic investigations: a short list of candidates can reduce both false positives and the processing time to identify an offender. The findings summarised in this paper can serve as a starting point for professionals in academia or industry involved in the field of forensic research. Finally, the deployment of hand recognition systems in authentication applications should benefit from the use of various security modules, e.g., PAD subsystems to spot attack presentations at the capture device and template protection schemes to prevent identity or demographic information theft.