3D Human Facial Traits’ Analysis for Ethnicity Recognition Using Deep Learning

Analyzing the human face has a long tradition starting from ancient populations basing on their observed descriptive facial attributes and using the natural ability of human visual system perception. However, the term “Facial morphology” is commonly used to describe studying the human facial form, shape, and structure. To analyze the facial details, many face recognition directions have been suggested and implemented for various medical and social purposes. Psychology, Neurology, in addition to sociology specialists in the 20th century highlighted essential questions about the human ability to identify faces. Visual assessment or facial appearance evaluation is the process of determining a human's identity or class according to his outward facial appearance. Anthropometric measurements are used to precisely measure the human face and recognize his identity. For example, skin tone, hair color, and eye color are prevalent traits examined visually with an acceptable degree of objectivity in human face perception procedures. However, 2D imaging faced difficulties and limitations due to many factors such as the resolution impact, the techniques of capturing, head pose variations, and illumination changes. The emergence of 3D imaging techniques such as Stereophotogrammetry and laser scanners, allows obtaining real 3D facial images including their natural color and texture details, in addition to the volumetric and depth information of the captured images. In addition to providing identity clues, the human face also conveys more demographic data such as reflecting their ethnicity, age, and gender (aka soft biometrics that influence applications of face recognition, security surveillance, and indexing facial data). Such research efforts led to a remarkable achievement in face attribute recognition using both 2D and 3D models, especially with the growing Machine Learning (ML) and deep learning (DL) abilities.

An anatomical study Zhuang et al. [1] showed that geometrical features also contain abundant information on soft biometrics. Early works such as Bertillon and McClaughry [2] created the first personal identification system for security identification using three groups of characteristics: (1) anthropometric, the measurements describe the height and length of the body parts, (2) morphological, the descriptors of the outer shape (such as anomalies of the fingers), and appearance (such as eye color), and (3) peculiar marks, the distinguishing signs seen on the body such as scars and moles. Heckathorn et al. [3] proposed a combination of personal characteristics or soft biometrics, such as race, gender, eye color, and several other obvious marks, to be utilized to accurately detect the identity. For instance, considerable differences in face dimensions, and lip distances among different ethnic groups of workers from the United States are concluded by Zhuang et al. [4]. These statistics showed that African-American individuals have distinguishable face and lip lengths from other ethnic groups such as; Caucasians, and Hispanics. In addition, the Caucasian face width differs from both African Americans and Hispanics. Zhuang et al. [5] collected anthropometric measurements from the individuals' faces. The author observed that females can be distinguished from males by ten dimensionality features. Related to ethnic groups, the author observed that the face length feature has a significant impact on identifying the African-Americans and Hispanics races since they have longer faces than Caucasians. Han and Choi [6] showed that Africans and Americans have shallower and smaller noses compared to Caucasians by a length difference of approximately two mm. Rebar et al. [7] in 2004, noticed that Caucasians can be distinguished by their long narrow faces for both males and females. Ballihi et al. [8] extracted radial curves and several specific geometrical circulars using the AdaBoost algorithm for gender classification on the FRGCv2 dataset. Another combination of 3D faces’ geodesic path features is examined by Abbas et al. [9] to implement a gender classifier on fifteen-year-old individuals from the ALSPAC medical dataset. The author observed that the nose morphology highly impacts identifying the face, especially the paths on Cupid’s bow, nose ala, and the inner eye canthi. Recent research by Mohammad and Al-Ani [10] showed that attribute-based biometric systems such as ethnicity are strong supporters of explicit essential biometrics in face recognition applications.

1.1 3D facial feature descriptors and feature extraction

Typically, 3D face recognition systems include multiple steps starting with the initial capturing using devices such as; multiple cameras and laser scanners. The obtained raw faces are then preprocessed by detecting the face area, denoising, cropping, reconstructing the missing part and holes, aligning it with other faces, and correcting the 3D pose to easily match it with the targeted or templates. Noise always inevitably affects most 3D scans captured by different acquisition devices to some degree. The general first step of denoising is usually done by cropping the face region due to its ability to eliminate irrelevant face data such as neck area and hair mineral details. Then, to describe the 3D faces in lower memory space, the important and descriptive features such as edges, corners, Gabor features, HOG features, and more are detected and extracted to be saved in vectors and matrices. In addition, the interest points such as landmarking or key points are determined manually or automatically using key point detectors such as ICP, SIFT, and other types to facilitate extracting the local features, calculating distances between the interest points, and calculating the sizes of face traits. A postprocessing step is also done to select the most important features that can cause changing the class of the face while they are changing in addition to advanced dimensionality reduction to reduce the resulted feature vectors and matrices using statistical methods such as PCA. The final step in face recognition is to explore the similarity among faces and match the extracted features between templates and targeted faces from the unseen input or from specific datasets using the similarity measures and the abilities of machine learning and supervised classification approaches such as SVM and recent deep learning such as CNN and pre-trained networks. Several common main steps of 3D face recognition are summarized in Table 1.

3D face recognition also can cope the uncontrolled illumination situations since it does not solely rely on pixel-level brightness while comparing the faces. Automatic face segmentation is also one of the new advantages obtained by 3D recognition since the image background can be typically synthesized separately in the reconstruction process.

Table 1. Commonly phases in 3D face classification

To simplify using the 3D features, another direction of research speeds up the features comparison by reducing the 3D features comparison into a 2D profile contour similarity measure between faces. Statistical approaches such as Principal Component Analysis (PCA), and Linear Discriminant Analysis (LDA), and that are have been used to decrease memory consumption by using a smaller-size descriptive feature. The well-known and most common PCA then has been extended to 3D feature reduction due to its high performance and simplicity with a limitation of image quality dependency [11]. A multimodal PCA approach is used by Tsalakanidou et al. [12] to enhance the recognition accuracy while implementing the proposed work in a 40-subject experiment. A comparing study by Chang et al. [13] used PCA analysis to extract 3D features over a long period and achieved near-optimal while using both 2D and 3D recognition. To solve the expression and head pose variation, many authors tried to describe the facial surface in a way that has more robustness for the isometric deformations caused by multiple expressions or head angles of the uncontrolled capturing environments.

Furtherly, maintaining the Geodesic Distances across 3D facial surfaces was another important emerged enhancing in face recognition and classification. In 3D feature extraction, researchers also tried to define point sets or profiles of each 3D face scan using surface-based global or local extraction algorithms from the initial whole 3D face features extracted by the traditional algorithms. Hybrid algorithms (using both local and global geometric surface information) also have been proposed with a limitation of complex defining of point sets. Geometrical approaches extract geometric features from 3D mesh form, including profiles, curvature, radial, iso level curves, 3D landmarks-based geodesic distance, symmetry properties, and correspondence vectors to describe the human face abstractly. The feature extraction step is often followed by the classification steps. In other words, the classification framework needs to extract facial features initially to build a good classifier including obtaining features such as demographic features, Gabor features, binary features, biologically inspired features, and local binary patterns, in addition to analyzing the extracted features using the abilities of statistical approaches and machine learning (SVM, LDA, PCA, random forest, and CNN deep learning). The main performance aspects of face descriptors include feature matching speed, discriminability power, and consumed memory. Researchers mentioned that the discriminative power needs to be balanced by computation complexity and extraction speed. A survey by Bellavia and Colombo [14] concluded that local SIFT-like descriptors can significantly compress the length and the matching time through their proper schemes. 3D key points are interesting points in shape that are found using certain geometric data from the face’s surface.

The three steps of the keypoints-based approaches are typically interest points detection, feature extraction (or description), and face features matching. The major expert directions to find the interesting point within the 3D face are the landmarks based on facial traits and the salient points on the facial surface. These salient key points strategies include those that use the well-known SIFT and their extensions. Eye corners and nose tips are two commonly used landmarks highlighted by researchers in local approaches due to their robustness to the expression variation. Several other Local Feature-based approaches are summarized in Figure 1.

11.png

Figure 1. General steps in 3D face recognition

1.1.1 SIFT-like local descriptors

In region-based face recognition algorithms, landmarking is a key component of feature space manipulation. The success of feature extraction will be closely correlated with the precision of the landmarking. Scale Invariant Feature Transform (SIFT) was introduced to extract local features with an enhanced robustness to pose variations, it also showed a good capability in handling the partially occluded facial parts. The SIFT feature descriptor showed good robustness to uncontrolled illumination, uniform change in scaling and orientation, partial noise, and distortion. The SIFT compares the source image with different resolutions and with specific changes in the scale during computing descriptors, it avoids the traditional limitation of holistic face representations because it can select the features that are not affected by scaling, noise, and rotation. In addition to its use as a feature descriptor, SIFT can be considered a feature extraction method due to its ability to reduce the image description into a set of points used to detect similar patterns in other images. Thus, it is widely used in image matching and object detection.

As described by Lowe [15], the process of extracting SIFT features involves four key steps: (1) selecting scale-space peaks, (2) localizing the keypoints, (3) assigning orientations, and (4) creating keypoint descriptors. Initially, interest points are detected by examining the image at various locations and scales. This is achieved by building a Gaussian pyramid and locating local peaks (known as keypoints) in a sequence of difference-of-Gaussian (DoG) images. In the subsequent step, potential keypoints are pinpointed with sub-pixel precision and discarded if deemed unreliable. Then defines the orientations per each point according to its local surrounding area. The assigned orientation(s), scale and local SIFT are also key point detectors due to their ability to detect the extreme points in different scale spaces, then calculate the directions around each key point according to the attributes of the facial features, to finally calculate the local feature descriptors.

Patch-SIFT [16] is a SIFT extension example proposed to extract key point features in small patches. Mian et al. [17] identified the key points and calculated the tensor features around them, then used the features fusion at the decision level to obtain the SIFT key points from both the depth image and 3D mesh. A 3D partial face matching is proposed by Berretti et al. [18] by extracting the features from specific spherical regions of different radiuses around each detected point. The author showed that the details in small regions of the face can be better captured while using a small spherical region radius. Although implementing the SIFT feature detector on 3D images that have depth information showed its sensitivity to significant pose variations, SIFT key points have been extracted directly from 3D mesh (Mesh-SIFT) to gain more robustness. Smeets et al. [19] used slant angles and histograms of shape index, in addition to Gaussian weight to construct the feature vector. Smeets’s results showed the Mish-SIFT robustness to the change in face expression, occluded and missing data, and outliers data with high recognition rates on both the FRGC v2 and the Bosphorus datasets. Lin et al. [20] calculated the third-order feature tensor per each detected salient point, and implemented the Voronoi diagram subdivision to augment the data to examine the intra and extra-face similarity using the pre-trained ResNet network. This similarity-based approach detected the salient points on the 3D mesh with an accuracy of 99.71% and 96.2% on the Bosphorus and BU3DFE databases respectively. Berretti et al. [21] used SIFT to highlight the feature points of depth images and extract specific curves combining the detected points. The authors posed side scans from the UND dataset and frontal faces from FRGC v2.0 to examine the SIFT robustness against the pose variation and detect the identical faces between these two datasets that provided multiple pose scans for each individual. Zhang et al. [22] used the shape index extrema as interesting points on the face surface. The author proposed a scale change procedure to eliminate a set of unstable key points initially, then extracted the features.

Darom and Keller [23] proposed a spin-based SIFT extension on the 3D mesh by measuring the vicinity of each detected keypoint using the depth map. The author enhanced the face rotation invariance by calculating the dominant angle using PCA. The resulting local features were experimentally robust to the scale variation and suitable for local matching of mesh segments. Additionally, proposed the local depth SIFT (LD-SIFT) to allow extending the SIFT extraction efficiently to the 3D meshes, and augment an angle estimation scheme to make the LD-SIFT a rotation invariant descriptor. Shi et al. [24] concluded that the SIFT-extracted feature points can be translated, and scaled and can show a rotation invariance too. The author also noted that the number and positions of these detected points are typically random, and don’t ensure that key points can be found in all face regions. Thus, the author proposed the ISIFT algorithm to enhance the robustness against facial expression variation.

1.1.2 Histogram of oriented gradients (HOG)

Using Histogram of Oriented Gradients (HOG) as a feature extractor on 3D facial meshes for face classification and recognition has shown promising results in various studies. HOG is a popular feature description commonly implemented in 2D models, but its adaptation to 3D facial meshes has demonstrated its potential in capturing distinctive facial characteristics. The advantages of using HOG in 3D facial mesh classification and recognition such as; its robustness to Pose Variations, and its efficient and simple implementation compared to other feature extraction methods. Thus, HOG is a suitable extraction for real-time and large-scale face classification and recognition tasks. Additionally, HOG allows focusing on capturing local shape information and texture patterns in a facial region, which makes it effective in highlighting distinctive features, such as edges, corners, and facial landmarks, which are crucial for accurate face classification and recognition. However, several challenges of using HOG for 3D facial mesh classification and recognition are also highlighted in academia such as vulnerability to occlusions due to its dependency on local gradient information, and its limited ability to capture global features due to its focus on capturing the local information.

1.1.3 Geodesic distance

The ability to extract facial curvature information from 3D imagery makes it easier to analyze the facial surface. In 3D face recognition and identification, image segmentation, and landmark localization, curvature information became a crucial part. The fact that the principle facial curvatures remain unchanged as the surface rotates or translates makes them extremely robust for feature space manipulation such as feature detection and extraction. Additionally, it has been demonstrated that principle curvature directions considerably improve identification performance because they have a strong potential to provide discriminative characteristics. Geodesic distance vector is a curve-based descriptor used as a shape descriptor that is invariant under the face shape isometric deformations. A geodesic distance matrix is a symmetric matrix whose elements (gi;j) are the geodesic distance between the points i and j. The GD vector associated with the points or landmarks will then be used as the shape descriptor for face classification and identification purposes. The geodesic path represents the shortest line between the reference point (or reference vertex on 3D mesh) which is usually considered as the nose tip, and another endpoint on the face surface. The algorithm visits all other endpoints and constructs a matrix of these calculated distances from the initial point. Then, it calculates the least distance from the start point to all other points. The resulting geodesic features matrix usually has a high dimensionality and needs to be reduced in size to allow easier analysis and machine learning phase in the next steps of face recognition and classification systems. With 3D Mesh facial form, Geodesic is usually preceded by meshes’ preprocessing to clean and remove noise, outliers, and artifacts. In addition to some mesh smoothing, hole filling, and vertex alignment to ensure the quality of the data. Then, the Geodesic descriptor calculates the geodesic distance for each vertex on the face mesh using algorithms such as Dijkstra's algorithm or Fast Marching Method due to their ability to calculate the shortest path between specific pair of points of the facial mesh. Once the geodesic distance for each vertex is calculated, the facial curvature features can be extracted. A common approach to extracting the geodesic features is local features extraction in which the algorithm computes local geodesic descriptors around each vertex. This can be done by considering the geodesic distances of neighboring vertices within a certain radius. These descriptors capture local shape information and can be used for tasks like facial expression recognition or landmark detection. Considering the isometric deformation nature of human facial parts while expression is changed, based on the various surface components used in face matching, Zhao et al. [25] separated the methods into three categories: (1) Iso-metric surface method, which matches the face with the complete isometric surface. (2) The iso-geodesic stripes method, which matches faces using a set of uniformly widened iso-geodesic stripes. 3) The geodesic-based approach, also known as the iso-geodesic method, uses the geodesic curves as the facial feature for face matching. The 3D face representation such as triangle mesh allows the description of each point with its depth value over the mesh surface and the local variation on the facial surface has been extracted by selecting the rigid regions of the face to identify the individuals. This representation allows calculating multiple geometric attributes such as geodesic distances, facial shape angles, and Gaussian curvatures. Drira et al. [26] investigated the 3D geometric analysis of human nose shape under multiple expressions of faces from the FRGC dataset and measured the geodesic distances between nasal surface pairs (a set of their radial nasal curves are transformed into the shape space for comparing the faces and calculating their similarity level). The author showed that the geodesic-based recognition outperforms the baseline ICP algorithm while multiple facial expression was used. Kurtek and Drira [27] developed a framework on the BU-3DFE dataset for analyzing the elastic form of hemispherical surfaces and explored the impact of parameterization-invariant, and elastic Riemannian metrics. The scans have been registered, deformed, and compared in this work, in addition to calculating the covariance, the PCA components, and analyzing the face symmetry. Lee and Krim [28] measured the shortest distances from a reference to points lying on a specific surrounding curve using Geodesic. The author then measured the similarity after extracting the deformed curves and used the Fourier transform to reduce the dimensionality of the feature space. The author measured the geodesic circle via the closed curve around the nose tip and tracked the boundary of the upper and lower lips. Berretti et al. [29] partitioned the facial surface into eight iso-geodesic stripes, and arcs between pairs of iso-geodesic stripes, and calculated their weight to obtain the relative spatial displacement.

1.2 Human Race identification based on face morphology

Human Face explicitly provides information for evaluating implicit data such as age, race, and gender. Face recognition methods generally can be categorized into local and holistic methods. Although many works are conducted using the holistic approach due to its simplicity, the local approach can be a more reliable and accurate recognition technique in applications focusing on specific parts of the human face.

The human race classification emerged as one of the most prominent face classification directions due to its omni-relevance with many social cognitive and specific perceptual tasks (emotion, belief, diseases, and more). The term “Race” is used to reflect the person’s physical appearance or characteristics regarding the main human categories of the population around the world. The population is categorized into seven large distinct populations or racial groups that are most commonly encountered and accepted racial groups. These seven main classes were the most accepted they represent more than 95% of the population.

2.png

Figure 2. Several faces from large distinct groups [30]

A commonly accepted racial categorization in Fu et al. [30] includes East Asian, African/African American, Native American/American Indian, Caucasian, Pacific Islander, Hispanic/Latino, and Asian Indian (See Figure 2). Literature shows that facial features with different relative positions have a close relationship with human ethnicity and facial landmarks and key points on specific positions of the face can reflect the positions of important descriptive areas and facial features that usually impact the resulting class of individuals. While automatic ethnicity classification is a useful starting point for facial analysis applications, most ethnicity classification techniques may necessitate a laborious feature extraction and model training procedure. Initially, psychologists approached the study of human face ethnicity and gender recognition from the standpoint of cognitive science. At that time, the characteristics of a person's face and the data they extracted, such as age, gender, and ethnicity, were referred to as soft biometric features. The term ‘ethnicity’ is identified as a soft biometric that describes belonging the individual to a particular social group having a common nationality, cultural tradition, or geographical factors. Thus, a group of humans contains people who have similar attributes such as language, religion, and nationality. The race classification might also categorize the population into smaller sub-ethnic groups such as; east Asian, Chinese, Korean, Japanese, and Indian. In addition to traditionally reported issues of face classification, many Issues and challenges are also reported in race recognition and classification and considered as challenging specific face trait recognition research areas. Issues such as; Intra-race, inter-race, and mixed-race are discussed in several works, and a few attempts are implemented by researchers to analyze the face variation across sub-ethnic groups (Turkish, Japanese, Spanish, …). Inter-race challenges of people belonging to two racial groups with a shared skin tone and mixed race challenges of People with average looks (such as; Native American, and Mexican-American are examples of race classification challenging issues [30]. However, the task of race classification might be partially enhanced by considering facial part information extraction and training the classifier on these extracted features and parts [31]. Gen Association with Regionalized Facial Features is reported by Richmond et al. [32] as described in Figure 3.

3.png

Figure 3. Facial features reported by Richmond

Because there aren't many large-scale 3D face datasets it was once thought to be extremely difficult to train discriminative deep features for 3D face recognition in comparison to 2D face datasets. To address this issue scientists attempted to modify a small number of 3D face datasets for 3D surface matching using the already-trained 2D face model. The proposed networks specifically designed for 3D face recognition and the recent capabilities of deep learning CNN encourage researchers to train massive volumes of 3D facial scans of identities more quickly and accurately. The recently proposed algorithms demonstrated excellent accuracy in both closed and open-world recognition scenarios based on the 3D datasets. The global approach component-based approach and hybrid approach - the three categories of feature extraction techniques for face recognition - are encountered once more in three-dimensional methods. Compared to 2D approaches, 3D race classification can provide better accuracy due to its robustness to the changing texture, skin tone, and appearance, in addition to its flexibility to correct the head pose and avoid the impacts of obstacles via 3D rotation.

1.3 Deep learning in 3D facial traits recognition

Many computer vision applications including 3D facial race recognition have been transformed by deep learning. Specifically useful for complex tasks like race classification from 3D facial data this technique uses multiple-layer neural networks to automatically learn relevant features from raw data. By allowing the model to learn hierarchical representations straight from the raw data, deep learning provides a considerable enhancement. The first step of the procedure could be data preprocessing which involves normalizing and formatting 3D face scans into the appropriate formats. 3D convolutional networks (3DCNNs) and CNNs approaches have been adapted to exploit the geometrical information found in 3D data in forms that hold the geometric information. The multi-layered models that extract features at various levels of abstraction gradually allow learning to distinguish between subtle yet discriminative features that may be difficult to identify using conventional techniques. This involves recording variances in the depth curvature and contours of the face that may be specific to various racial groups. The model becomes adept at recognizing patterns that are important for race classification, even when these patterns are not explicitly defined. Deep learning has been more successful in 3D facial race recognition thanks to the availability of large-scale datasets with a variety of 3D facial scans from different races. With the help of these datasets, models may be trained to identify a person's race from their facial traits and to generalize effectively to new data. Though incredibly successful, deep learning-based 3D facial race recognition is not without its difficulties. While 2D image datasets are more abundant annotated 3D facial datasets are still scarce. Thus, deep learning has allowed models to learn directly from raw data greatly improving the accuracy and dependability of 3D facial race recognition. This approach has proven effective in capturing intricate facial features that are critical for distinguishing races, and it continues to drive progress in the field of facial recognition technology. AlBdairi et al. [33] developed a DCNN-based 2D race recognition approach to determine the human race, the author used high-performance devices to build a a face recognition system and proposed a technique called field-programmable gate arrays (FPGAs). The periocular area is analyzed for race and gender by Khellat-Kihel et al. [34] who proved that deep learning techniques in race prediction still require a large amount of labeled data, and accordingly proposed a DCNN-based predictor to solve several specific biometrics issues on the periocular part of the human faces. Periocular regions’ features are extracted from 2D faces using different pre-trained architectures such as Alex-net and ResNet-50. Fan et al. [35] compared five forms of 3D data including depth images, normal maps, point clouds, DAE, and HHA (obtained from FRGC v.2 and BU-3DFE datasets). The author proposed a data augmentation approach by synthesizing multi-view 3D faces using a DAE-based deep learning model. Xia et al. [36] proposed a correlation-based 3D facial shape model, and highlighted salient features extracted from three traits, and then used the RF algorithm for ethnicity estimation on the FRGC-v2 dataset. The study comes to a conclusion demonstrating ethnicity is considerably correlated in the 3D representation of the human face.

1.4 3D face analysis for identifying the race, a literature review

Lu et al. [37] used both registered range images (describe the surface shape) and intensity images for race classification (See Figure 4). The author used an SVM classifier to classify the faces and achieved 96.8% accuracy. A mixture of two frontal 3D face databases is used for evaluating the proposed schemes. Toderici et al. [38] suggested retrieving subjects from deformed 3D meshes (only global shape information) through a metric function (Harr wavelet and CW-SSIM, or structure similarity).

The author proposes four classifiers: wavelet-based learning, multi-dimensional scaling (MDS), kernelized KNN, and KNN. The FRGC v2.0 dataset was used to test these classifiers, and they produced mean accuracy values of about 99 percent. Toderici et al. [38] showed that the estimated probabilities can serve as feature importance scores, and used Gauss-Markov posterior marginals to classify the Asian and White people from the BU-3DFE and FRGC v1 tables. To prove his idea, the author compared his achieved accuracy levels with the levels achieved by Ocegueda et al. [39]. Zhong et al. [40] combined the use of 3D shape and the Boosted Local Texture) to enhance the race-based classification. To extract the local texture, the author used the Oriented Gradient Maps (OGMs) to highlight ethnicity-related local texture and used Adaboost to add an associated weight for individuals. Experiments are carried out on the FRGC-v2 dataset to categorize Asians and non-Asians with an accuracy of 98.3%.

4.png

Figure 4. Integration scheme of both range and intensity [35]

Table 2. Several existing face analysis works for race classification

Zhong et al. [40] computed the visual codes for the eastern and western individuals using Gabor features and the learned visual codebook (LVC) method. Next, the merging and mapping distances were measured (by calculating the max distance function). The author suggested a "membership function" in the second level to determine the membership degree of the western and eastern groups for the faces taken from the FGRC v2 dataset.

Berretti et al. [29] modeled the three-dimensional faces of multiple stripes centered at the tip of the nose, in which most of the points of each stripe remain within the same stripe even when the facial expression changes. Nine stripes, each measuring one centimeter in width, were employed by the author to distinguish between various individuals. The author additionally divided each stripe into three sections—lower (L), upper-left (UL), and upper-right (UR)—concerning the coordinates of the nose tip to account for the expression deformation. This work implemented the suggested measuring on the FRGC-v2 dataset (measuring the spatial displacement between iso-geodesic stripes) and the SHREC 2008 dataset (measuring the inter-stripe and intra-stripe 3DWW) distances using the 3D Weighted Walkthroughs (3DWW) approach. Several existing related works are summarized in (Table 2) with their findings and the supported races.

1.5 Problem statement and the contribution

Although existing race classification works tried different approaches to extract the face features, insufficient research efforts have been proposed with a focus on analyzing the 3D mesh form for identifying the human race. The size limitation of the publicly available 3D facial datasets in a mesh form, the unfair racial distribution, capturing cost, and privacy aspects, are factors that negatively impacted this research direction. The data limitation forced a need for proposing efficient extractions of geometric and shape features from the well-known mesh representation, and for building accurate race classifiers that can cope with the limited data training challenge.

The context of this work is to analyze the shape of the 3D facial scans of real individuals after extracting four types of 3D facial features implemented on different facial areas. It focuses on the mesh representation of 3D faces due to its shape details and geometrical surface information, and due to the limitations of traditional 3D representation such as depth images and z-coordinate approximations. The texture analysis, the 3D representations that had limitations in flexibility and in describing the surface geometry, and the reconstructed synthesis faces, are out of the scope of this work. The 3D face analysis of this work is extended to compare the used feature extractions for race classification due to its challenging fine and salient variation in facial details. This work is also aided by several experimental and knowledge-based secondary phases of preprocessing and 3D mesh registration that have a direct impact on the extraction and classification results. In addition, Deep Learning is used in this work for automatically landmarking and extracting the distance-based features according to specific interest key points on the facial mesh. The results of this work highlight suitable feature extraction for race classification by several important extractions, these comparing results also highlight the facial areas and traits that could highly impact identifying the human race by the face shape details.

The proposed classifiers of this work are trained on the extracted features in the previous phase and evaluated according to their accuracy and validation calculations of the used ML classifier. The collected SIFT features from the cropped frontal face are used to train the NN and SVM classifiers. The experiment is run using a single Core-i7 CPU, 16 G of random-access memory, and a Windows 10 operating system. Three main racial categories have been used; African, Chinese, and White. In the second and third experiments, the HOG and Geodesic features have been extracted from the cropped face area for classifying the same three races. Additionally, a local feature-based experiment is investigated to measure the classification performance while training the algorithm on just cropped mouth parts of Headspace facial scans to explore the feasibility of using the mouth area in identifying the race. The Mouth-SIFT features showed a high accuracy (100%) in addition to decreasing the feature space size. Comparing the obtained results showed that the use of the SIFT features on the complete facial area of the Headspace dataset outperforms both the HOG and Geodesic performance with an accuracy of 90%. While the HOG-based classification achieved the least accuracy (70%) in distinguishing the three used classes. A comparison of the achieved NN classifier accuracy of the used extractions is shown in Figure 12. The results of the SVM classifier are shown in Figure 13. For further evaluation, the confusion matrix parameters have been used to evaluate the positive and negative errors of the actual and the predicted class labels as shown in Figure 14.

12.png

Figure 12. Comparing the achieved NN classifier accuracies

13.png

Figure 13. Comparing the achieved SVM classifier accuracies

14.png

Figure 14. Confusion matrix parameters of the NN classifier

Extraction of the SIFT features from the cropped mouth area enhanced the prediction accuracy in addition to reducing the memory consumption by reducing the consumed time and the search area of the used extraction and reducing the size of the resulting feature vectors. The high performance of 3D SIFT was consistent with the literature of traditional 2D works that highlight SIFT's ability to capture the local facial details with high robustness against noise and scale variation. In addition, SIFT descriptors outperform the HOG and Geodesic in the feature vectors’ size due to their role in reducing the image description into a relatively acceptable size set of points used later to measure the similarity in the patterns of other scans in the dataset.