Machine Learning for Masked Face Recognition in COVID-19 Pandemic Situation

Recognition systems plays a key role in recognizing the person for different purposes. These recognition systems include biometric system, speech recognition system, graphics recognition system, character recognition system, image recognition system and facial recognition system. The biometric system is mainly used in security systems to get access. This system is used to allow the person to access the system by recognizing the texture of their fingers. When the user places a finger on the biometric plate the system matches their fingerprint with the recorded fingerprints in the database. On the other hand, the speech recognition system is used to break the audio of speech recordings into individual voices, to recognize the languages, to fit the most related word in that language and to create the print ready dictation. There are different algorithms used by copying these sounds into the text.

A few decades ago, face recognition was an important topic for the researchers. But it is no longer an important topic for researchers because COVID-19 causes all humans to wear face masks. It is important to wear a face mask to prevent the spread of the virus. However, previous facial recognition systems have failed to recognize the faces in COVID-19 epidemic due to the presence of face masks on human faces. The face mask covers half of the human face. Thus, face recognition systems are not able to recognize the faces properly. To solve this problem, we have proposed a masked face recognition system using two machine learning algorithms i.e., SVM and RF based recognition system. The main objective of the proposed masked face recognition is to efficiently recognize the masked face. The dataset generated are of 28 different class with 1470 number of training and testing images.

In this paper, the SVM and RF based recognizers are used to recognize the masked face images. The recognizer detects the masked image using SURF features and find out the similarities between these features. These features are input to the recognizer structure and the figure of merit is average recognition accuracy. The average recognition accuracy is approximately 97% with SVM and with the random forest, the average recognition accuracy is 98.2% for 28 class problem.

The remainder of the paper is organized as follows: Section II presents the comprehensive literature review of existing face recognition techniques. Section III presents the methodology of extraction of various features and the classifier structure. Simulation results are presented with detailed discussion in section IV. Conclusions and future work are finally stated in the section V.

1.png

Figure 1. Block diagram of proposed system model

The block diagram of proposed system model is presented in Figure 1. The system model for recognition of masked face is basically categorized into three phases: data collection, training of recognizer and the testing of recognizer. In the first phase, masked face image has been gathered and dataset is generated. Secondly, using the dataset, recognizer system is trained by using support vector machine and random forest which is also termed as training of the recognizer. Third phase is the face recognition, which is the last stage and called testing of the classifier.

3.1 Data set collection

The data for the recognition problem is gathered in the form of images. Each image is of same size and dimension. Images are captured of a person from different angles. The dataset includes masked and unmasked face images of every person. The collected data set contains 1470 images for the recognizer training and testing. All the images are labelled with their name, institution, and profession. The size of the dataset is 800x600x3 and total number of elements in each image is 1440000. The dataset consists of 28 classes and precise picture of the dataset is shown in Figure 2. The dataset and their detailed labelling are shown in the Table 1.

3.2 Feature extraction

The speeded up robust features (SURF) are extracted from the image dataset. SURF indicates, a speedy and robust algorithm. It is used to represent the similarity and comparison between the different images. The main aim of SURF is to track or recognize the object.

In the proposed system, SURF is used to recognize the features of human faces and then compare the similarities between the different images. After the extraction, the clustering is used to segregate images having similar features.

1. Choose the number of classes which are used to initialize the center points.
2. The center points are of the same length and known as vectors.
3. The distance is computed between the point and the center of each image.
4. The result is determined based on the closest center.
5. All the above steps are repeated for several iterations, finally, the results are presented.

2.png

Figure 2. Dataset for masked face recognition

3.3 Recognizer algorithm

In this research, support vector machine and random forest are employed as a recognizer, to recognize the masked face. The recognizer algorithm is discussed as following:

Table 1. Recognition dataset with class labelling

3.3.1 SVM based recognizer

SVM is a machine learning algorithm which is used for the recognition/classification purposes. A technique used in the SVM algorithm known as kernel trick. It is a function that converts low dimensional space into a high dimensional space. It separates the dataset according to the labels and then gives an output after recognizing the face. The SVMs are used to extract the features from the image by using kernel tricks [17]. The kernel tricks (K_n) are used to find out the similarities (S_n) between the image given to the system and the image stored in the dataset as shown in the Figure 3. According to these similarities the system will recognize the masked person. The block diagram of the proposed SVM based recognizer is shown in Figure 4.

3.png

Figure 3. Mathematical model of SVM based recognizer

4.png

Figure 4. Block diagram of SVM based recognizer

A face recognition system is proposed to recognize faces with and without the face masks. To solve the masked face recognition issues, we have used SVM algorithm. There are two stages of the SVM algorithm. In the first stage, the images are collected from the gallery and reading all the features of each face using SVM. However, the system is trained using SVM and results are stored in the dataset. The second is the testing stage in which the features will be tested using the classifier. The first step in SVM is to import and explore the dataset. After the dataset is pre-processed and split into two parts i.e., attributes and labels. The pre-processed and split data is divided into training and testing sets. Furthermore, predictions are made after training the SVM algorithm. The results are finalized with the help of predictions.

Kernel trick “K” is a dot product between two functions. The kernel SVM maximizes the hyper plane margin and the transformation formed by it is higher in dimensional space. Let the dataset is $X=\left\{\left(x_{1}, d_{1}\right)\left(x_{2}, d_{2}\right) \ldots \ldots\left(x_{m}, d_{m}\right)\right\}$, where $x_{i} \varepsilon \mathcal{R}^{n}$ and $d_{i} \varepsilon\{-1,+1\}$, x_i and d_i are representing the data points which are of class type. The optimal hyper plane of SVM is as follows:

$\mathrm{w}^{\mathrm{T}} \mathrm{x}_{\mathrm{i}}-\mathrm{b} \geq+1, \quad \mathrm{~d}_{\mathrm{i}}=+1$       (1)

$\mathrm{w}^{\mathrm{T}} \mathrm{x}_{\mathrm{i}}-\mathrm{b} \leq-1, \quad \mathrm{~d}_{\mathrm{i}}=-1$       (2)

where, the weight coefficients of the hyper plane are represented by w and b represents the bias network. The minimization problem is as follows: $\min _{\mathrm{w}, \mathrm{b}} \frac{1}{2}\|\mathrm{w}\|^{2}$ subjected to constraints $\mathrm{d}_{\mathrm{i}}\left(\mathrm{w}^{\mathrm{T}} \mathrm{x}_{\mathrm{i}}-\mathrm{b}\right) \geq 1$. The optimization problem solved by Lagrangian dual and dual problem is given below:

$\begin{aligned} \min _{\mathrm{w}, \mathrm{b}} \mathrm{L}(\mathrm{w}, \mathrm{b}, \mathrm{v}) &=\frac{1}{2}\|\mathrm{w}\|^{2}-\sum_{\mathrm{i}=1}^{\mathrm{m}} \mathrm{v}_{\mathrm{i}}\left[\mathrm{d}_{\mathrm{i}}\left(\mathrm{w}^{\mathrm{T}} \mathrm{x}_{\mathrm{i}}-\mathrm{b}\right)\right.\\ &-1] \end{aligned}$       (3)

After the quadratic programming is applied, the v_i values and decision of the class is:

$f(x)=\operatorname{sign}\left(w^{T} x-b\right)=\operatorname{sign}\left\{\sum_{i=1}^{m} v_{i} d_{i} x_{i}^{T} x_{i}-b\right\}$       (4)

which corresponds the non-zeros v_i are support vectors. The pseudo code of the proposed recognizer based on SVM is given in Table 2 and Table 3.

Table 2. Extract features from the DATASET

Table 3. SVM recognizer

3.3.2 Random forest based recognizer

Random forest is an algorithm which is used for the machine learning processes. As the name shown, i.e., random forest, that the forest is made up of the trees. Therefore, the samples are taken from the dataset and then form a tree which predicts the result. The result is finalized based on most voted prediction. Figure 5 depicts the flow of random forest algorithm according to the proposed system. In the first step, each image from the dataset is taken as a sample for training. Secondly, for each sample the algorithm will construct a decision tree. From each decision tree we will get the prediction result. Next, from every predicted result, the voting will be made. Lastly, the final predicted result is selected from the most voted prediction.

5.png

Figure 5. Generalized random forest for face recognition

The start of random forest is to select features k from total number of features m. Therefore, it is observed that the features and observations are taken randomly in the image. In the second step, using the best split point, the randomly selected k features are used to find the root node. Thirdly, the daughter nodes are calculated using the best split point. From first three steps the tree with a root node is formed and gets the target as the leaf node. Lastly, 1-4 steps are repeated to randomly create n number of trees. The trees created randomly formed the random forest. In random forest algorithm, the generalization error is presented in the equation given below:

$G_{e}=\rho_{x, y}(\operatorname{mar}(x, y)<0)$       (5)

where, the generalized error is represented by $G_{e}$, the landslide conditioning factors are x, y which indicates over the x & y space with the margin function mar. Therefore, the margin function mar is shown in the equation below:

mar(x,y)= ȃѴ_m ᴊ(ḩ_m (x)=y)-max(i≠j) ȃѴ_m ᴊ(ḩ_m(x)=i)       (6)

where, ḩ_m are the classifiers and ᴊ(.) is the indicator function. The steps for masked face recognition using random forest are given in Table 4.

Table 4. RF based recognizer