Integration of Face and Gait Recognition via Transfer Learning: A Multiscale Biometric Identification Approach

The field of biometric recognition systems, utilizing facial and gait data, has been the subject of extensive scholarly exploration. A number of studies have proposed the combination of these two biometrics, aiming to enhance the accuracy and robustness of person identification.

Zakaria et al. [18] proposed a Convolutional Neural Network (CNN) consisting of 15 layers. The efficacy of this CNN was assessed using the YouTube Faces database and was subsequently compared to the Principal Component Analysis (PCA) and FHMM methods utilizing the ORL face database.

In a different study, Tabassum et al. [19] applied Discrete Wavelet Transform (DWT) coupled with PCA for feature extraction. They proposed a fusion of the CNN results using detection probability entropy and a fuzzy system, achieving an identification rate of 89.56% in the worst-case scenario and 93.34% in the best case.

A novel approach was implemented by Mehmood et al. [20], who proposed Human Gait Recognition (HGR) from various viewing angles. Their methodology involved the use of the pre-trained Densenet-201 CNN model for feature extraction, feature reduction based on a mixed selection method, and learning through four major stages of supervised learning techniques. The crucial stage was the extraction of CNN features, with the most active features being simultaneously combined from the second and third layers. Subsequently, the Firefly algorithm and skew-based technology were employed for feature selection. They achieved an accuracy of 94.7% at an angle of 180 degrees.

Mogan et al. [21] proposed a hybrid model merging multilayer perceptrons with the pre-trained DenseNet-201 for the CAS, OU-ISIR D, and OU-ISIR databases. Their approach involved extracting the gait energy image, obtaining representative features, and applying transfer learning from the pre-trained DenseNet-201 model. A multilayer perceptron was used to identify correlations between these features, which were then assigned to appropriate class labels by a classification layer, achieving an accuracy of 92.22%.

In a study by Liu and Liu [22], they used the CASIA-B gait dataset and the UCMP-GAIT dataset. Their Two-Stream neural network (TS-Net) model concurrently extracted dynamic deep features from gait images and static invariant features from multiple resolutions. The goal of person recognition was transformed into a binary classification problem through similarity learning techniques, achieving an accuracy of 92.22%.

Wang and Yan [23] proposed a method based on convolutional long short-term memory (Conv-LSTM) and applied it to the OU-ISIR LP and CASIA datasets. They introduced a modification of Gait Energy Images (GEI), referred to as frame-by-frame GEI (ff-GEI), to increase the available gait data and relax the restrictions of gait cycle segmentation. Their study demonstrated the effectiveness of ff-GEI by analysing the cross-covariance of a single person's gait data, achieving a Correct Recognition Rate (CRR) average of 95.9%.

Aung and Pluempitiwiriyawej [24] applied a deep convolutional neural network (CNN) technique for gait biometric person identification, using Gait Energy Images (GEI) of individuals, and utilized the CASIA-B gait dataset. Their empirical findings indicated superior recognition efficacy when compared to other advanced machine learning models.

Punyani et al. [25] introduced "double-level fusion" that combined data and output levels. PCA and MLG were applied to the OU-ISIR Gait, Large Population, and USF Gait datasets to reduce the dimensionality arising from integrating facial features with gait. The KNN algorithm was utilized for recognition. Their results showed that the Mean Absolute Error (MAE) reached 6.11.

Rahman et al. [26] configured a biometric system combining facial and gait biometrics at two levels (rank and score level fusion). The Histogram of Oriented Gradients (HOG) method was applied for facial feature extraction, while structural features were used for gait feature extraction. Their highest fusion result was 96.67% for logistic regression rank-level fusion.

Aung et al. [27] proposed the use of a CNN algorithm with transfer learning to extract gait and facial features. They applied the feature-level fusion method to merge facial and gait features, and then applied various classification algorithms, achieving the highest accuracy of 97.3% using the logistic regression (OvR) algorithm.

In the study by Maity et al. [28], a system was proposed for accurate multimodal human identification from low-resolution video surveillance footage, using a single biometric data source for low-resolution face and frontal gait recognition. The Adaboost detector was used for automatic detection of low-resolution face images, while a quick object segmentation algorithm was used to segment frontal gait binary silhouettes. The low-resolution face images were pre-processedusing super-resolution techniques to generate a high-resolution representation. This was followed by normalization of lighting and poses and image synthesis via registration. Gabor and Local Binary Patterns (LBP) features were then extracted from the synthetic face images. The nearest neighbor classifier was employed to accomplish rank-1 recognition for each modality. Subsequently, score-level fusion was used to combine the results of each separate recognition process. Their results showed that the combined use of low-resolution face and frontal gait modalities led to the highest rank-1 recognition accuracy, compared to the performance of each modality independently.

Manssor et al. [29] employed the YOLO-face algorithm along with the Eigenfaces approach for recognizing faces. For gait recognition, they applied the YOLO algorithm in tandem with Hidden Markov Models (HMMs). The outputs of the face and gait recognition algorithms were combined using decision-level fusion. The YOLOv3-Human model was trained using the DHU Night, FLIR, and KAIST databases.

In summary, these studies provide a comprehensive overview of the advancements and methodologies utilized in the field of biometric recognition systems, specifically focusing on facial and gait data. The diverse approaches employed in these studies have contributed to enhancing the accuracy and robustness of person identification systems, thereby paving the way for future research in this domain.

A framework has been proposed to identify humans using the face and gait with several stages. In the first stage, a dataset of the face and gait is created for the same people, where two digital cameras were used (to record) video clips. The first camera (the front camera) photographed the person as he passed from the front, and the second (recorded) video clips from a side view of the person for Recognition of the gait. The viewing area of the front and side cameras was ten meters, and 10-second video clips were recorded. The experiment was conducted at the Duhok Technical Institute and the Shekhan Technical Institute / Polytechnic University of Duhok for 65 volunteers. In the second stage, the videos for each person were divided into two videos, the first for the face and the second for the gait. After creating the videos, the face and gait videos were processed separately. To extract facial features, the Haar Cascade algorithm was used to track and detect facial images from front camera videos and performed image pre-processing that included standardization of size, centering, and rotation according to the position of the face and using 100 facial images for each person. After pre-processing, pre-training algorithms were applied to the face images using (inception_v3 and DenseNet201) algorithms. They extracted the features specific to the face and the number of faces particular to each person. As for gait and extracting gait features, the background was initially isolated from the person's body using the MOG2 algorithm to convert the video clip of the gait to white as the person's body and a black background. Then, the HOG algorithm will track the person's gait within the video clip. The result is a set of sequential cut images of the gait with sizes according to the distance and proximity of the person from the camera’s position. At this stage, image pre-processing was used to standardize the sizes and focus the images. The GEI gait energy finding technique was used to convert gait images into distinguishable images, which collects chromatic units between images and adopts a new gait energy image for each image. The same facial feature extraction algorithms were used in the final stage of feature extraction. As shown in Figure 3.

3.jpg

Figure 3. Suggested methodology for face and gait recognition

After obtaining the facial and gait features, three sets of features (face, gait, face, and gait) are established. This stage is called (features-level fusion). After the feature extraction phase, the data set was divided into (80:20) a training set and a test set. The selected machine learning algorithms are trained for classification and person identification. The algorithms (SVC, KNN) were trained on the training data set for face once, gait energy once, facial features, and gait once, testing the models using test and comparison data and determining the most accurate model for each case. In the last stage, the outputs of the selected models are combined with a meta-classifier to create a model that combines the biometric features at the model output level, called (score-level fusion) and testing the model using the test data.

4.1 Create a dataset for face and gait

65 volunteers from the Polytechnic Institute in Dohuk –Information Technology Department were used to create a data set to distinguish humans by face and gait. Two digital cameras with a resolution of 108 megapixels and a recording space of 10 * 7 meters were installed. A special camera for recording video clips of the face was installed at the end. Recording space While the gait recording camera was placed in the center of the recording space to take the complete steps of the gait. As shown in Figure 4.

4.jpg

Figure 4. Recording space environment

Humans were passed in the recording space at a rate of 10 seconds per person and a size of 75 megabytes per video clip, and videos are stored in MP4 format. Experiments were conducted in the natural environment of the institute's corridors. Table 1 shows a description of the recorded video clips.

Table 1. Description of the recorded video clips

4.2 Feature extraction

To integrate more than one biometric feature at the level of features, the gait, and face must be detected and tracked as an initial stage for the recognition system to work.

4.2.1 Face images detection and tracking

Haar Cascade is an Object Detection Algorithm that detects faces in images and real-time videos. "haar" refers to a rectangle-shaped mathematical function [30]. Haar is a single rectangular wavelet (one high and one low interval). It features one bright and one dark side in two dimensions (2D) [31]. The purpose of cascade classification is to incorporate additional features efficiently. Initially, haar's image processing only depends on each pixel’s RGB value, after which the picture is processed in rectangle forms with specific pixels in each shape. Each form is analyzed, and the limit level (threshold) is reached, revealing the dark and bright areas [32]. The haar feature formula states that if the average value of the result is more than the threshold, the haar feature exists. To train the algorithm, a large number of positive photos with faces and a large number of negative images with no faces are provided. The pixels with values 1 are darker in the haar feature, whereas those with 0 are brighter. Each is in charge of identifying a particular feature in the picture: an edge, a line, or any other structure where the intensities abruptly shift [33]. It is important to note that the training results have been verified using a significant amount.

5.jpg

Figure 5. Extracting face images using Haarcascade method

Figure 5 shows that the learning algorithm's definition of a set of Haar-like characteristics makes up each cascade level. In the first phases, classifiers are trained to recognize almost all candidates resembling faces while rejecting most negative sub-windows. This design can significantly speed up the discovery process because most negatives may be eliminated during the first two or three stages. Most computing resources are concentrated on face-like sub-windows. Stage classifiers systematically assess each sub-window, combining the results into the stage for each Haar-like feature. To assess if the present sub-window is a face-like filter, all the characteristics at a location are computed, and the aggregated value is compared with the stage boundary value. Only if the preceding step is active is the succeeding stage triggered.

4.2.2 Gait images detecting and tracking

The stage of extracting gait energy images goes through several stages, starting with background subtraction, converting the video clip to a black background, and representing the person walking in white color using the MOG2 algorithm. In the second stage, the rectangle of the gait image is detected and converted into a gait image for each frame in the gait video. This process is done using HOG. This stage is followed by calculating gait energy and creating a gait energy dataset. This study used the function createBackgroundSubtractorMOG2 provided in the OpenCV package for Background subtraction. Figure 6 shows the gait segments before and after background subtraction.

6.jpg

Figure 6. Background subtraction for gait image

To track and extract images of the person's gait HOGDescriptor() function was used. The default human detector size is 64×128, so the size of the person who wants to detect it must be at least 64×128. Figure 7 shows the images extracted from the video clip.

7.jpg

Figure 7. Extracting gait image

Then, the set SVMDetector function was used to set the Support Vector Machine to pre-train the gait detector and load it with the cv2.HOGDescriptor_get Default People Detector() function. Another important function used is DiscoverMultiScale(). This function implements a detection phenomenon through a multi-scale window. To extract the gait energy, the center of mass of the image must be determined before it is collected. p=(i; j) represents the pixel value in row i and column j. To find the gait center of mass, the brightest area must be determined, and it can be calculated by taking the average value of x and y and determining the place with the most density. Eqs. (1), (2), and (3) can be used to find d and, $\hat{y}$ which are the coordinates of the gait center of mass.

$\hat{x}=\frac{\sum_{i=0}^m\,\,\sum_{j=0}^n j\,\,*\,\,p(i, j)}{r}$                 (1)

$\hat{y}=\frac{\sum_{i=0}^m\,\,\sum_{j=0}^n i\,\,*\,\,p(i, j)}{r}$     (2)

$r=\sum_{i=0}^m\,\,\sum_{j=0}^n p(i, j)$         (3)

The calculation method of each feature, when the brightness value at the pixel (x, y) of the frame of the normalized silhouette sequence is cap I. open paren x, y, t, close paren, And the walking cycle in P, is as follows. GEI is created by finding the average of the silhouette series within one walking cycle:

$G E I(x, y)=\frac{1}{p}\,\,\sum_{t=1}^p I(x, y, t)$          (4)

In GEI, the moving part becomes smaller by averaging the brightness values, so the pixel's shade expresses the movement's magnitude, and the part with less movement is expressed brightly.

As shown in Figure 8, the gait energy was found for every four frames in the video clip, and the gait energy was extracted as an image for recognition.

8.jpg

Figure 8. Gait energy image

4.3 Feature extraction using transfer learning

In this study, transfer learning is adopted, whereby a pre-trained model can be used as a feature extractor for any image of visual objects classified in computer vision. Transfer learning is a machine learning method that uses a pre-trained neural network. Two types of pre-trained deep learning algorithms (inception_v3 and DenseNet201) were used to extract gait and facial features for identification purposes.

4.3.1 Feature extraction using Inception_v3

Inception-v3 is a deep neural network, and the network consists of 11 units starting with five types in total. A pre-trained model was used to extract gait and facial features. The trained model was used in Python and Keras deep learning framework. Table 2 determines the algorithm's parameters and values.

Table 2. Inception_v3 pre-trained parameters

The input to the algorithm was 100 images of both gait and face energy. RGB images with a size of 128 * 128 were used. The output of the mixed10 layer is adopted by the InceptionV3 algorithm. (2, 2, 2048) were extracted, equivalent to 8,192 features for each image for each image of faces and gait energy.

4.3.2 Feature extraction using DenseNet201

DenseNet201 is a deep neural network. This paper used a pre-trained model to extract gait and facial features. The Conv5 Block32 Concat layer was selected from the DenseNet model as the output layer. The trained model was used in Python and Keras deep learning framework. The parameters and values of the DenseNet201 algorithm were determined in Table 3.

Table 3. Densenet201 pre-trained parameters

The input to the algorithm was 100 images of both gait energy and face. RGB images with a size of 128 * 128 were used. The DenseNet201 algorithm adopts the output of the Conv5 Block32 layer. (4, 4, 1920) were extracted features for each image for each image of faces and gait energy

4.4 Facial and gait fusion

The merger is done at one of two levels (features-level fusion and score-level fusion) to identify humans using multiple biometric features. Two models (SVC and KNN) were used to model the data and compare them for the first level, and for the second level, decision trees were used to combine the classifiers’ outputs. Experiments were conducted using Python to build classification models within the Sklearn framework.

4.4.1 Recognize using features-level fusion

The combination of feature-level information provides more information about the person to be verified. The proposed feature level fusion combines pre-match biometric information of the face and gait. Feature-level merging is not widely used because it is difficult to integrate incompatible feature vectors from multiple methods. Therefore, in this study, it was proposed to use gait energy images with images of faces and to use the same feature extraction algorithms and create two datasets that are identical in terms of face and gait composition. The simplest form of feature-level fusion is the collection of data sets at the feature level. This type of merging doubles the number of distinctive features.

As in Figure 9, two biometric features of a person were combined to distinguish face and gait energy. In the first stage, facial features and gait energy were extracted using pre-training algorithms (inception_v3 and DenseNet201). The output of this stage is two datasets for extracted features; then, the two feature datasets are combined to create a new dataset that integrates facial features and gait energy. In the next stage, the selected machine learning models are trained to develop a classifier model.

9.jpg

Figure 9. Block diagram of feature-level fusion

4.4.2 Recognize using score-level fusion

In the case of score-level fusion, the outputs of the classification models are merged using a supporting model. This method enables to collect the predictive values of several classifiers and give a final result. In this paper, as in the case of features-level fusion, facial features, and gait energy were extracted using deep learning, and two data sets were created, as in Figure 10.

10.jpg

Figure 10. Block diagram of the score-level fusion

After extracting the facial features and gait energy, two models for classification are trained. The first model is trained on features extracted from the face, while the second model is trained on features extracted for gait energy. In the last stage, a final model is trained on the outputs of the two models.

In this study, a gait and face dataset has been created to create a multi-biometric recognition system to create an accurate recognition system using security system recorders. The data was pre-processed, and the faces and gait images were extracted. The study concluded that using more than one biometric feature increases recognition accuracy and reliability. Using transfer learning and feature fusion at the feature fusion level to integrate the dataset and then feed it into a machine learning model is a promising solution in biometric feature integration. Achieve the best model in feature extraction (inception_v3) with SVC when combining face and gait at the feature level, followed by the same algorithms for face and gait and using decision trees as a support classifier. In future work, more than one data set will be taken, and a more comprehensive range of deep learning and machine learning algorithms will be used.