Face Mask Segmentation Method Combining Salient Features and Gender Constraints

To realize real-time facial detection on hardware platforms, this paper optimizes the multi-task cascade convolutional neural network structure by introducing depthwise separable convolution. Depthwise separable convolution is a computationally efficient and effective convolution method that separates spatial convolution and channel convolution, achieving lower parameter and computational complexity while still effectively extracting image features. This enables the algorithm to achieve higher recognition accuracy at a lower computational cost in the facial detection task with the fusion of landmark features and gender constraints. The reduction of model parameters makes it easier to deploy the model on hardware platforms, such as embedded devices and mobile devices. This allows the facial detection method with the fusion of landmark features and gender constraints to be widely applied on various hardware platforms. The multi-task cascade convolutional neural network structure adopted in this paper can handle multiple tasks simultaneously, such as mask detection and gender recognition. This allows the algorithm to achieve the recognition of multiple attributes in the facial detection task with the fusion of landmark features and gender constraints, improving the comprehensiveness of recognition.

The following formula gives the calculation of the parameter amount M_t and M_d in standard convolution, assuming the side length of the convolution kernel is represented by C_G, and the input and output channel numbers are represented by D_in and D_out, respectively:

$M_t=C_G \times C_G \times D_{i n} \times D_{\text {out }}$                      (1)

$M_d=F_{\text {out }} \times Q_{\text {out }} \times D_{\text {out }} \times C_G \times C_G \times D_{\text {in }}$                     (2)

Figure 1 shows the calculation process of Mt and Md in standard convolution. In contrast, the formula for the parameter amount M_t and computation amount M_d in DepthWis of the adopted depthwise separable convolution is as follows:

$M_t^{D W}=C_G \times C_G \times D_{i n}$                      (3)

$M_d^{D W}=C_G \times C_G \times D_{\text {in }} \times F_{\text {out }} \times Q_{\text {out }}$                     (4)

The following formula gives the calculation of the parameter amount and the computation amount in PointWise:

$M_t^{P W}=1 \times 1 \times D_{\text {in }} \times D_{\text {out }}$                     (5)

$M_d^{P W}=F_{o u t} \times Q_{\text {out }} \times 1 \times 1 \times D_{\text {in }}$                     (6)

By adding the above formulas in series, the total parameter amount and the total calculation of the depthwise separable convolution can be obtained, and the calculation process is given by the following formula:

$M_t=M_t^{D W}+M_t^{P W}$                     (7)

$M_d=M_d^{D W}+M_d^{P W}$                     (8)

From the above derivation process, it can be seen that depthwise separable convolution significantly compresses the model parameter amount and computation amount based on traditional standard convolution. Assuming the ratio of the total parameter amount of depthwise separable convolution to standard convolution is represented by β, and the ratio of the total computation amount is represented by γ, the following calculation formula is obtained:

$\beta=\frac{M^{C Q}+T Q}{M}=\frac{C_G \times C_G \times D_{\text {in }}+D_{\text {in }} \times D_{\text {out }}}{C_G \times C_G \times D_{\text {in }} \times D_{\text {out }}}=\frac{C_G^2+D_{\text {out }}}{C_G^2 \times D_{\text {out }}}$                     (9)

$\gamma=\frac{M^{D W}+T Q}{M}=\frac{C_G^2+D_{\text {out }}}{D_{\text {out }} \times C_G^2}=\frac{1}{D_{\text {out }}}=\frac{1}{C_G^2}$                    (10)

Training facial landmark features and gender attributes separately requires the separate deployment of facial landmark feature recognition networks and gender recognition networks. The disadvantage of this approach is that hardware needs to read the weights of both networks, which doubles the overall parameter amount compared to a single network. Therefore, this paper designs a facial multi-attribute recognition network with the fusion of landmark features and gender constraints using VGG-11 as the backbone network. By integrating the landmark features and gender constraints into a single network, the total parameter amount can be reduced. This can ease the burden on hardware and improve computational efficiency. Integrating landmark features and gender constraints into a single network also allows for better utilization of shared feature representations within the network. This helps to improve the performance of multi-attribute recognition tasks and avoid redundancy due to separate training. Additionally, the VGG-11 network, compared to other deeper convolutional neural networks (such as VGG-16, VGG-19, etc.), has a lower computational complexity, which can maintain high recognition accuracy while reducing runtime.

To further reduce the parameter amount and obtain a smaller input size, this paper provides an optimization scheme for the original VGG-11 network. Figure 2 shows the optimized network structure. By removing the Conv8, FC, and FC2 layers, the model's parameter amount is significantly reduced, easing the burden on hardware resources, reducing computational costs, and increasing computation speed. Adjusting the input image size to 224×224 can reduce the computation amount and memory requirements. This allows the network to achieve high performance even under lower computational resource conditions. The optimized network structure is more compact, facilitating improved model deployment flexibility on different devices. By adjusting the size of the FC3 layer to 1×1×22, the network can adapt to facial multi-attribute recognition tasks. This enables the network to handle multiple tasks simultaneously, such as landmark features and gender constraints, enhancing the comprehensiveness and efficiency of recognition.

This paper attempts to combine facial salient features and gender in a single network structure for output. The specific representation and combination method are given below: Suppose there are N salient features (such as keypoints, facial expressions, etc.), which can be numbered in order from 1 to N. For gender, we can use the same encoding method as described above, i.e., 0 for females and N+1 for males. Next, the facial salient feature values and gender values are added together to obtain a (N+1)*2-dimensional output space. In the last fully connected layer of the VGG network, the output size is adjusted to 1×1×((N+1)*2), and the probability values are obtained through softmax. In the output processing phase, the maximum probability value can be found from the (N+1)*2 probability values and the corresponding salient feature and gender classification results can be calculated. For example, suppose there are 10 salient features, so the gender encoding is 0 (female) and 1 (male), resulting in 22 outputs with a size of 1×1×22. By processing the output probability values, a classification result containing both salient features and gender attributes can be obtained, and the specific calculation method is shown in the following formula. Assuming that the network's probability of predicting females is represented by T_FE, the network's probability of predicting males is represented by T_MA, and the network's probability of predicting age l is represented by T_age(l), then:

$T_{F E}=\sum_{i=1}^{11} T_i$                    (11)

$T_{M A}=\sum_{i=12}^{22} T_i$                    (12)

$T_{a g e}(l)=T_l+T_{l+11}$                    (13)

In this paper, N salient features are numbered from 1 to N, and gender is encoded using 0 (female) and N+1 (male). Next, the facial salient feature values and gender values are added together to obtain a (N+1)*2-dimensional output space. For females (gender value 0), the probability of having a salient feature value of i is the probability of the i-th output of the softmax layer. For males (gender value N+1), the probability of having a salient feature value of i is the probability of the (N+1+i)-th output of the softmax layer. Therefore, the total probability of having a facial salient feature value of i is the sum of the probabilities of the i-th output and the (N+1+i)-th output of the softmax layer. To obtain the probability values for all facial salient feature classifications, this process can be applied to all salient feature values (1 to N). First, initialize a probability value array P of length N. For each salient feature value, calculate P[i]=softmax_output[i]+softmax_output [N+1+i]. Finally, the array P will contain the probability values for all facial salient feature classifications.

In the optimization of the network, the gender loss function and facial salient feature loss function are cross-entropy loss function and L1 loss function, respectively. In the facial salient feature classification task, this paper treats facial salient feature classification as a probability distribution problem, using Gaussian distribution to represent the probability distribution of facial salient features, and introducing prior knowledge, i.e., the pre-set Gaussian distribution, through the constraint of KL divergence. By comparing the predicted facial salient feature probability distribution of the model with the pre-set Gaussian distribution, the difference between the two distributions is minimized to improve the prediction accuracy. At the same time, the KL divergence loss function has good interpretability, making the performance of the model during the prediction process easier to understand and evaluate. Assuming that the true distribution is represented by t(a), the predicted distribution is represented by w(a), and the matching degree between the predicted distribution and the true distribution is represented by C_KL, the following formula gives the calculation formula of KL divergence:

$C_{K L}(t \| w)=\sum_{i=1}^M t\left(a_i\right) \log \left(\frac{t\left(a_i\right)}{w\left(a_i\right)}\right)$                    (14)

Assuming that L1 loss function is represented by LOSS₁, KL divergence loss function is represented by LOSS₂, and cross-entropy loss function is represented by LOSS₃. The loss function of the constructed multi-attribute recognition network can be calculated through the following formula:

$L O S S=0.4 \operatorname{LOSS}_1+0.6 \mathrm{LOSS}_2+x \mathrm{LOSS}_3$                    (15)

11.png

Figure 1. The calculation process of M_t and M_d in standard convolution

2.png

Figure 2. Optimized network structure

Observing the results of the saturation component grayscale histograms and brightness component grayscale histograms shown in Figures 4 and 5, we can draw the following conclusions. In the saturation component grayscale histogram, the vast majority of pixel points are concentrated between 135 and 140, indicating that the saturation of the face mask area is relatively low and the color is relatively uniform. In the brightness component grayscale histogram, the vast majority of pixel points are concentrated between 0 and 50, with virtually no pixel points distributed in other areas. This suggests that the distribution of the face mask area in the H component is relatively narrow, with little color variation. Combining these observations, we can analyze that by combining the suppression algorithm and the improved gray world algorithm, the color difference between the face mask area and other areas can be enhanced while maintaining the original texture and edge information, thus improving the accuracy and robustness of face mask segmentation.

Furthermore, this paper conducted a comparative analysis of the experimental results of face detection method optimization strategies, as shown in Table 1. The table shows the impact of various optimization strategies on feature extraction accuracy, gender recognition accuracy, mIoU, and pAcc. The original VGG-11 network performs well in feature extraction accuracy, gender recognition accuracy, and other aspects, but there is still room for improvement. Compared to the original VGG-11 network, deleting Conv8, FC, and FC2 layers improves gender recognition accuracy and mIoU but slightly reduces feature extraction accuracy and pAcc. This suggests that reducing network complexity to some extent helps with gender recognition and segmentation performance. Adjusting the input image size in this optimization strategy significantly improves gender recognition accuracy and pAcc, but slightly reduces feature extraction accuracy and mIoU. This indicates that adjusting the input image size contributes to improving the accuracy of gender recognition. Adjusting the size of the FC3 layer significantly improves feature extraction accuracy and gender recognition accuracy, and also increases mIoU and pAcc. This demonstrates that adjusting the size of the FC3 layer can effectively improve network performance. Combining salient features with gender improves feature extraction accuracy, gender recognition accuracy, mIoU, and pAcc. This proves that combining salient features with gender helps improve network performance. Compared to the original VGG-11 network, the final model shows significant improvements in feature extraction accuracy, gender recognition accuracy, mIoU, and pAcc. This demonstrates that the integration of the aforementioned optimization strategies can effectively improve the performance of face detection methods. By comparing various optimization strategies and their results, the conclusion can be drawn that the face detection method optimization strategy combining salient features and gender constraints effectively improves feature extraction accuracy, gender recognition accuracy, mIoU, and pAcc, thus achieving more accurate face detection.

4.png

Figure 4. Histogram of saturation component in face mask area

5.png

Figure 5. Histogram of brightness component in face mask area

Table 1. Comparative analysis of experimental results of face detection method optimization strategies

Table 2. Loss function experiment results of the multi-attribute recognition network

Table 3. Comparison of experimental results of different face mask region compensation methods

Table 4. Comparison of experimental results of different face mask region segmentation methods (mloU)

Table 2 presents the experimental results of the loss functions for the multi-attribute recognition network. From the table, we can see the impact of various loss functions on feature extraction accuracy, gender recognition accuracy, mIoU, and pAcc. The original VGG-11 network performs well in terms of feature extraction accuracy, gender recognition accuracy, etc., but there is still room for improvement. Compared with the original VGG-11 network, introducing the L1 loss function strategy significantly improves feature extraction accuracy, mIoU, and pAcc, but slightly decreases gender recognition accuracy. This indicates that the L1 loss function can improve the accuracy of feature extraction, but has little effect on gender recognition. Introducing the cross-entropy loss function significantly improves feature extraction accuracy and gender recognition accuracy, but slightly decreases mIoU and pAcc. This suggests that the cross-entropy loss function has a significant impact on feature extraction and gender recognition. Introducing the KL divergence loss function significantly improves feature extraction accuracy, gender recognition accuracy, and pAcc, and also increases mIoU. This indicates that the KL divergence loss function can further improve the accuracy of feature extraction and gender recognition. By comparing various loss functions and their results, we can draw the conclusion: in the constructed multi-attribute recognition network, introducing L1 loss function, cross-entropy loss function, and KL divergence loss function all contribute to improving feature extraction accuracy and gender recognition accuracy. In particular, introducing the KL divergence loss function has the most significant improvement on overall model performance. Therefore, the multi-attribute recognition network with these loss functions has high effectiveness in improving recognition accuracy.

When building a dataset for face mask segmentation, it is essential to ensure the dataset is diverse and representative to train a model with high generalization ability. This paper constructs a public scenario dataset and a specific scenario dataset. When building the dataset, make sure the sample size is sufficient and balanced among categories to avoid training bias caused by class imbalance. At the same time, when dividing the training set, validation set, and test set, ensure that the sample distribution in each subset is roughly the same to more accurately assess model performance. If the face mask segmentation model needs to be applied in a specific industry, more relevant scene photos can be included in the dataset to train a more targeted model.

Figure 6 shows the loss accuracy curve of network training. It can be seen that the constructed network model converges quickly during training and tends to be stable in the later stage. The accuracy of facial salient feature and gender attribute recognition reaches 92% and later exceeds 96%. These indicators show that the network model has good performance. Combining this information, it can be verified that the network model constructed in this paper has high accuracy in face mask segmentation tasks based on salient feature and gender constraint fusion. This indicates that the model can effectively recognize facial salient features and gender attributes, thereby improving the accuracy of face mask segmentation. The model converges quickly during training and is stable in the later stage, indicating that the model can learn effective feature representations during training, thus providing better prediction results in the testing phase.

According to Table 3, the experimental results of different face mask region compensation methods on two sample sets can be seen. It can be seen that in the two sample sets, the combined algorithm adopted in this paper achieves the best segmentation effect, with mloU of 0.914 and 0.967 respectively. For other methods, the gray world algorithm, bilateral filtering and inverse projection perform relatively well on the two sample sets. This shows that these methods have a certain robustness in handling different types of sample sets. The Gaussian pyramid fusion and Laplacian pyramid fusion methods perform worse in sample set 1, but have a significant improvement in sample set 2. This may indicate that these two methods have better adaptability to specific types of images in sample set 2, but weaker generalization ability in sample set 1. Histogram equalization and color balance methods perform relatively stable on the two sample sets, but the overall effect is not as good as other methods. This shows that these methods may not have sufficient advantages for the task of face mask region segmentation.

According to Table 4, the experimental results of different face mask region segmentation methods on the two sample sets can be seen. In the two sample sets, the segmentation effect of the model proposed in this paper is the best, with mloU values of 0.8187 and 0.9525 respectively. Among other methods, fuzzy c-means clustering, edge detection and level set methods perform relatively well on the two sample sets. This shows that these methods have a certain robustness in handling different types of sample sets. Threshold segmentation, graph cut algorithm and U-Net perform worse in sample set 1 but have significant improvement in sample set2. This may indicate that these methods have better adaptability to specific types of images in sample set 2 but weaker generalization ability in sample set 1. Although the proposed model achieved good results on both sample sets, there is still room for further optimization to improve its applicability in different scenarios. In summary, face mask region segmentation experiments on different types of sample sets can clearly show that the proposed model has better performance and stronger generalization ability.

6.png

Figure 6. The curve of network loss and accuracy