Deep Learning Based Gender Identification Using Ear Images

Biometric identification technology is rapidly advancing, driven by its commendable security and reliability. As a consequence, it is finding widespread applications in various domains, including e-commerce, e-government, and crime detection. Biometric identification-based verification systems, such as finger-print recognition and facial identity verification systems, are constantly evolving [1, 2]. In recent years, automatic gender determination has attracted considerable interest due to its potential in a variety of applications, including human-computer interaction, banking transactions, disease diagnosis, visual surveillance, and demographic data collection. Gender determination is essential to biometric identification systems because it permits the database to be cut in half, thereby simplifying and expediting the identification process. This increases the efficacy and efficiency of these systems in their respective applications.

Ear recognition technology has emerged as a significant non-invasive personal identification method with numerous applications, comparable to facial recognition. The human ear has stable, well-structured characteristics that are unaffected by facial expressions and aging. Notably, the earlobe, a distinguishing characteristic utilized in forensic investigations, continues to change over time. The ear’s visibility in images recorded by security cameras and its ease of capture in profile views, video recordings, or photographs increase its identification utility [3].

Due to its unique and relatively stable characteristics, the ear functions as a valuable biometric identifier in both biometric research and forensic medicine [4]. In contrast to the face, which can be affected by factors such as facial expressions, facial hair, and cosmetics, the ear’s appearance is stable, making it an advantageous identifier [5]. Particularly, the earlobe stands out as a distinguishing characteristic frequently used in forensic investigations, and it continues to change over time, providing additional identification clues [6]. The ability of security cameras to capture the ear, whether it is partially or completely visible, aids in the identification process [4]. Moreover, the ear is simpler to capture than the face in profile views, video recordings, and photographs [7]. In Figure 1, the fundamental components of the human ear are provided.

Indeed, numerous studies have concentrated on the use of ear images for identification purposes [3-9]. In contrast, the research on extracting sensitive biometric features from ear images, such as age, has been relatively limited. Despite this, there has been considerable interest in using ear images for gender classification, as evidenced by several studies [10-13]. Previous research has investigated the use of ear images for gender classification, laying the groundwork for our research. In the study conducted by Gnanasivam and Muttan [10], they utilized ear hole measurements as a reference point and calculated Euclidean distances between the detected ear hole from masked ear images and seven distinct ear features. They used the Bayes classifier, the KNN classifier, and neural networks as classifiers. The KNN classifier yielded the best results, achieving a remarkable classification accuracy of 90.42%. Zhang and Wang [11] investigated gender classification using profile face images and ear images separately. They employed support vector machines (SVM) with a histogram intersection kernel for classification. By performing score-level fusion based on Bayesian analysis, they were able to improve accuracy. In the study conducted by Khorsandi and Abdel-Mottaleb [12], they employed Gabor filters for feature extraction and performed classification based on features extracted by dictionary learning. This approach resulted in an accuracy of 93.5%. The study utilized 2D images from the UND biometrics dataset collection F [14]. The fusion approach achieved an impressive accuracy of 97.65%, whereas the accuracy for face-only classification was 95.43% and ear-only classification was approximately 91.7%.

11.png

Figure 1. Morphological components of the human ear [6]

Despite the fact that these studies have demonstrated a high rate of accuracy for gender classification based on ear images, they have primarily relied on conventional machine-learning techniques. Our research extends beyond these prior works by proposing a novel hybrid recognition architecture, MobileNetV2 with ViT, for gender classification tasks. Unlike the traditional methods employed in prior studies, our model leverages deep learning techniques, particularly CNN and ViT, which have shown success in computer vision tasks. The combination of these models provides superior performance, computational efficiency, and reduced parameter count compared to individual approaches. Moreover, we provide t-SNE results to assess how well our proposed model separates gender-specific characteristics in a low-dimensional space. This analysis provides helpful insights into the interpretability and feature representations of the model, which can be used to better comprehend its decision-making process.

The main contributions of this work can be summarized as follows:

•   We propose a hybrid recognition architecture called MobileNetV2 with ViT, which combines the strengths of CNN and ViT, to perform gender classification tasks.

•   Recently, the efficacy of models like MLP-Mixer and ViT, which are alternatives to CNNs, has been evaluated. In terms of performance, computational efficiency, and the number of parameters, a hybrid model architecture that incorporates the benefits of these models has been identified as the best option.

•   We provided t-SNE results to observe how well the proposed model disentangles features compared to other models in a low-dimensional space.

The rest of the article is structured as follows: Section 2 pro-vides a brief overview of relevant prior research. Section 3 delves into the methodology and materials used in the study. The comparison results of the methods are presented in Section 4. Finally, the article concludes with potential future directions.

3.1 Dataset

In the field of ear recognition, datasets collected under unrestricted conditions are scarce. These datasets differ in terms of ear morphology, the number of individuals included, and the data collection techniques. One of the earliest datasets employed for ear recognition research is the WPUT dataset [27], which consists of 2071 ear images from 501 individuals of varying ages and includes variables such as illumination, head position, and occlusions. The AWE dataset [3] comprises 1000 ear images of 100 celebrities collected from the internet, whereas the UERC dataset [3] is an extension of AWE, containing 11,804 ear images from various individuals. The In-the-wild ear dataset [28] comprised of 2,058 ear images cropped from a larger dataset originally intended for face recognition, containing data from 231 individuals. On the other hand, the EarVN1.0 dataset [29] is one of the largest public ear datasets, containing 28,412 RGB ear images from 164 Asian individuals. This dataset was constructed by cropping Internet images of ears, capturing various camera and lighting conditions. The presence of pose, scale, and illumination variations in these datasets makes them suitable for building models adaptable to real-life scenarios; however, it also presents training challenges. We utilized the EarVN1.0 dataset for our research, which includes gender information. Figure 2 depicts example images of the ear from the EarVN1.0 dataset.

2.png

Figure 2. Sample ear images taken EarVN1.0 dataset; top: left ear images; down: right ear images for the same person. The images’ resolutions vary because they were captured under unconstrained conditions

3.2 Method

In this section, we aim to devise an architecture for the ear gender recognition problem that achieves both low-parameter complexity and state-of-the-art performance. To accomplish this, we propose a hybrid architecture that incorporates three ground-breaking deep learning techniques: CNN, MLP-Mixer, and ViT, as well as their derivatives.

3.2.1 Convolutional neural networks

Within the domain of CNN, the central operation is convolution. This process involves applying a KxK sized convolution kernel to an input image with dimensions HxW, where H and W represent the height and width of the feature map, respectively. The input image consists of N channels, and the terms M and N denote the number of convolution kernels and output feature-map channels, respectively. Figure 3 illustrates this process. As shown, standard convolution entails convolving the input data with multiple convolution kernels of the same depth, and the final result is computed by summing up the results corresponding to each channel. In recent times, CNN-based models that have demonstrated exceptional performance in the Large Scale Visual Recognition Challenge (ILSVRC) [30] have also proven to be effective in various other problem domains. Several examples of these models are described below.

3.png

Figure 3. The overall process of standard convolution [8]

3.2.2 AlexNet

The AlexNet model, proposed by Krizhevsky et al. [31], achieved significantly improved recognition accuracy on the ImageNet dataset [32], resulting in its victory in the ILSVRC-2012 competition. This model, which consists of eight layers, has demonstrated unique advantages in image classification tasks [33]. The data source requires input in the format of 227×227×3 pixels, where the dimensions 227×227 represent the height and width of the input image, and the value 3 indicates that the data is in RGB mode with three channels. The first two layers of the model perform convolution, followed by activation (using ReLU), max-pooling, and normalization operations. Subsequently, the output of the second layer undergoes convolution with 256 feature maps using a kernel size of 5×5, a stride of 1, and other parameters matching the first layer. The third and fourth layers only perform convolution and activation operations, while the fifth layer combines convolution, activation, and normalization operations. The output of the fifth layer is then reshaped and flattened into a long vector, which is fed into a traditional neural network comprising three fully connected layers. The first two fully connected layers contain 4,096 neurons each, and the final layer has output nodes corresponding to the number of classes, i.e., 1,000.

3.2.3 VGGNet

Researchers from the Oxford University Visual Geometry Group and Google DeepMind devised VGGNet, a deep CNN consisting of six models with varying depths, ranging from 11 to 19 layers [34]. The models with the greatest number of layers, namely 16 and 19, have proven to be the most effective for image classification and localization tasks. The architecture of VGGNet is based on five convolutional layers, each utilizing a kernel size of 3×3, a stride of 1, and a padding of 1, followed by a max-pooling layer with a size of 2×2 and a stride of 2. Subsequent to the final maxpooling layer, the features within the image feature map are integrated through three fully connected layers, with the final layer employing Softmax for image classification and normalization. In comparison to traditional CNNs, VGGNet’s significant contributions include a smaller size of convolution and pooling kernels, an increased number of convolutional layers, the use of pre-trained data for parameter initialization [35], and a method for converting fully connected layers into convolutional layers during the testing phase.

3.2.4 Inception

InceptionV1, also known as GoogLeNet, is a deep convolutional neural network architecture proposed by Szegedy et al. [36]. It achieved a remarkable accuracy of 93.3% in the ILSVRC competition and stood out for its significantly reduced number of parameters compared to earlier models like AlexNet and VGG.

The unique feature of this architecture is its departure from the conventional sequential process. Instead, it employs a combination of network layers, pooling layers, and parallel computations of both large and small convolutional layers. It also utilizes 1×1 convolutions for dimensionality reduction. This approach of parallelism and dimensionality reduction significantly reduces the number of parameters and computational cost, making it more memory and computation efficient. Various versions of Inception exist, including InceptionV1/GoogLeNet, InceptionV2, and InceptionV3, among others.

3.2.5 ResNet

ResNet, a deep learning architecture proposed by He et al. [37], addresses the challenges of training deep neural networks, which include high computational costs and limitations on the number of layers. ResNet tackles these issues by introducing skip-connections or shortcuts. Unlike other architectural models, ResNet’s performance does not degrade as the depth of the architecture increases, and it significantly improves computational efficiency, enabling better training of networks. The ResNet model incorporates skip-connections, ReLU, and batch normalization in its architectures, typically between two to three layers. The exceptional image classification performance of ResNet, as demonstrated by He and his colleagues [37], highlights its efficacy in extracting image features.

3.2.6 DenseNet

DenseNet, introduced by Huang et al. [38], is a deep learning architecture that employs direct connections between all layers, facilitating an efficient flow of information. Each layer in the DenseNet architecture receives input from all preceding layers and shares its feature maps with all subsequent layers. The feature maps generated by a given layer are concatenated with those from the preceding layer, leading to a design known as DenseNet.

3.2.7 MobileNet

MobileNet is a deep learning architecture designed to improve accuracy by minimizing the number of convolutional layers. This reduction can however cause the issue of gradient vanishing. MobileNetV2 was developed as an enhancement over MobileNetV1 to address this issue. MobileNetV2 incorporates the residual structure from ResNet to enable better information flow between layers and mitigate gradient vanishing during backward propagation. MobileNetV2’s fundamental building block is a depthwise separable convolution block with linear bottleneck and inverted residuals, which transforms features from N to M channels. The bottleneck consists of a 1×1 convolutional layer with a linear activation function, followed by a depthwise convolutional layer with subsampling using the s parameter. The network structure of MobileNetV2 consists of 19 layers and is depicted in Table 1, where conv2d represents standard convolution, avgpool represents average pooling, c denotes the number of output channels, and n signifies the number of repetitions. The intermediate layers are in charge of feature extraction, whereas the final layer is responsible for classification.

Table 1. The overall network structure of MobileNetV2, where 'k' signifies the number of classes

3.3 MLP-mixer

MLP-mixer [39]: With the help of state-of-the-art models, MLP-mixer offers a fairly straightforward architecture that performs competitively on benchmarks for image clasification. The MLP-mixer is focused only on multi-layer perceptrons (MLPs) that do not employ convolutions or self-attention. The Mixer layer is made up of two separate MLP layers: one that applies MLPs to individual picture patches independently, “mixing” location-specific characteristics and another that applies MLPs to many patches simultaneously, “mixing” spatial information. Only simple matrix multiplication operations, data layout adjustments (reshapes and transpositions), and scalar nonlinearity are used by Mixer.

MLP is a type of neural network architecture that process input data through various layers and produce output data as a result. The architecture of an MLP includes an input layer, several hidden layers, and an output layer. The hidden layers process the data and learn higher level features. The output layer perceptrons then produce the output data. An MLP includes weight and bias values for each perceptron. These values are learned during training and used to process the data. The weight values represent the connections between perceptrons and the bias values represent the threshold values of the perceptrons. This architecture is typically feed-forward, meaning that the data flows from input to output. It is also typically trained using the back-propagation algorithm, which updates the weight values to reduce the error rate between the output data and the real data. MLP is a powerful architecture that can be used for a variety of applications such as natural language processing, image recognition and audio recognition. With its ability to process data through multiple layers and learn advanced features, it has proven to be a valuable tool in the field of machine learning. An MLP can be represented mathematically using the following equation:

$\begin{gathered}y=f(W n * f(W n-1 * \ldots * f(W 2 * f(W 1 * \\ x+b 1)+b 2) \ldots+b n-1)+b n)\end{gathered}$              (1)

The equation, where y is the output of the Multi-layer Perceptron (MLP), x is the input data, f is the activation function (e.g., sigmoid, ReLU, etc.), W1, W2, ..., Wn are the weight matrices for each layer, b1, b2, ..., bn are the bias vectors for each layer and n is the number of layers in the MLP, describes how the input data is transformed through multiple layers of perceptrons. The input data is multiplied by the weight matrix of the first layer, passed through the activation function to introduce non-linearity, and then added to the bias vector of the first layer. This process is repeated for each subsequent layer, with the output of one layer serving as the input for the next layer. The final output is obtained by passing the output of the final hidden layer through the activation function and adding the bias vector of the output layer. It should be noted that the above equation depicts a feed- forward process, where information flows from the input layer to the output layer without feedback. The back-propagation algorithm can be utilized to adjust the weight and bias values during training to minimize the error between the predicted output and the actual output.

The MLP-mixer is a method that uses a sequence of linearly projected image patches, referred to as tokens, as input. The input data is arranged in a table with dimensions “patches x channels”. The mixer has two types of MLP layers to mix spatial information: channel-mixing MLPs and token-mixing MLPs. Channel-mixing MLPs allow communication between different channels by processing each token independently and using individual rows of the input table. Token-mixing MLPs facilitate communication between different spatial locations by independently processing each channel and using individual columns of the input table. By alternating between these two types of layers, the MLP-Mixer is able to effectively mix both input dimensions, resulting in mixed feature maps.

The MLP-mixer takes as input a sequence of S non-overlapping image patches, each projected to a hidden dimension of C. This produces a two-dimensional input table, X; ϵ; RSXC. The mixer is made up of multiple layers, each of the same size, and each layer consists of two blocks of MLPs. The first block is the token-mixing MLP, which operates on the columns of XT and maps the input to the same dimension as the output. The second block is the channel-mixing MLP, which operates on the rows of X and maps the input to the same dimension as the output. Each MLP block has two fully-connected layers and a nonlinearity operation. The nonlinearity is applied to each row of the input data tensor individually. In Table 2, we have provided the specifications used in the MLP-mixer architecture.

Table 2. Specifications of the mixer architectures

3.4 Vision transformer (ViT)

Vision Transformers, initially introduced by Dosovitskiy et al. [40], are a deep learning architecture that has demonstrated superior performance in image classification applications when trained on large-scale datasets compared to CNNs. However, their reliance on vast quantities of training data and computational resources poses a challenge. To address this issue, Touvron et al. [41] introduced a data-efficient version of ViT by using commonly used data augmentation and manipulation techniques for CNNs. They also improved the performance of ViT through a transformer-based teacher-student approach. The high performance of ViT has prompted further research into its use for various vision tasks [42].

ViT is a deep learning architecture designed to categorize images by modeling a series of image patches into a semantic label. Unlike traditional CNN designs, the ViT utilizes the encoder module of the transformer to allow for the interpretation of information throughout the entire image through its attention mechanism. The architecture of the ViT typically includes (1) an embedding layer, (2) an encoder, and (3) a final classifier head.

The first step in the process is to divide the training set images into non-overlapping patches. Each patch is evaluated as a separate token by the transformer. A [c, h, w] dimensional image results in n series of [c, p, p] patches, where c represents the number of channels, h represents the height, w represents the width, and p is the size of the patch. The number of patches, n, is calculated by dividing h w by p². Typically, a patch size of 16 or 32 is chosen, as a smaller patch size results in a wider array and vice versa.

3.4.1 Embedding layer

The patches obtained from dividing the image into non-overlapping sections are transformed into a 1-dimensional vector via a trainable linear projection (embedded matrix E) prior to being fed into the encoder. The embedded patches are then combined with a learnable embedding classification indicator, x_class, necessary for performing the classification task.

The position of each patch within the image is incorporated into its representation through the addition of position embeddings. The position embeddings, denoted by E_pos, have a dimension of (n+1)×D, where n is the number of patches and D is the dimension of the vector representation. The combined representation of each embedded patch and its position embedding is represented by z₀ in Eq. (2), with E being the embedding matrix of size (p₂c) and Epos$\in$R^(n+1)×D .

$z_0=\left[x_{\text {class }} x_p^1 E_i x_p^2 E_i \ldots ; x_p^n E\right]+E_{\text {pos }}$            (2)

3.4.2 Vision transformer encoder

The encoder in ViT is constructed from L identical layers, each layer comprised of a Multi-Head Self-Attention (MSA) block and a Multi-Layer Perceptron (MLP) block. These blocks are separated by a normalization layer, and there are skip connections present after each block. The MLP block is composed of two layers utilizing the GELU activation function.

$z_l^{\prime}=\operatorname{MSA}\left(L N\left(z_{l-1}\right)\right)+z_{l-1}, l=1 \ldots L$            (3)

$z_l=M L P L N z_l^{\prime}+z_k^{\prime} l=1 \ldots L$            (4)

$y=L N_L^0 z$            (5)

The first component in the sequence is taken from the last layer of the encoder and fed to the head classifier as shown in (5) to predict the class label.

3.4.3 Compact convolutional transformer (CCT)

ViT architectures have been demonstrated to achieve superior performance in image classification tasks when trained on large-scale datasets, however, they come with the requirement for significant amounts of data and computational resources [38]. To overcome this challenge, researchers have explored the combination of transformers and convolutions to benefit from their respective strengths. Hassani et al. [43] introduced the Compact Convolutional Transformer (CCT) architecture, which combines a patch-based approach to preserve local information, addressing some of the limitations of ViT, while still achieving improved performance. This model can encode the relationships between patches differently from the original ViT. CCTs are efficient to kenizers that preserve local spatial relationships while having short receptive fields. Additionally, the transformer encoder provides a sequential pooling approach called SeqPool which gathers sequential information from the encoder. SeqPool eliminates the need for an additional Classification Symbol. Figure 4 shows the entire end-to-end architecture of the model.

4.png

Figure 4. The proposed method for ear gender recognition

In this section, we discuss the steps we took to attain state-of-the-art results for the problem of ear gender recognition. Our objective is to develop a hybrid model by combining three approaches that have recently made significant advancements in image classification: CNNs, MLP-Mixer, and ViT. In this context, upon examination of the literature, no studies have been observed that used MLP-Mixer and ViT models, while many models such as VGG, ResNet, etc., that performed well on the ImageNet dataset have been used with CNN models. Nguyen-Quoc and Hoang [8] examined the performance of popular CNN models on the EarVN1.0 dataset, and the highest test accuracy of 91.12% was obtained with the VGG19 model. However, the VGG19 model is a dense network and contains approximately 137 million parameters (see Table 3). In contrast, the MobileNetV2 model has a modest number of parameters, approximately 2 million, but a performance of 85.55%. Comparing the training periods of the two models reveals that MobileNetV2 is trained approximately three times faster (see Table 5). Since our aim is to construct a model with few parameters, the MobileNetV2 model was chosen as the CNN model, and experimental studies were conducted to improve its performance.

Table 5. Comparison of model results in terms of test accuracy, run time, and parameter size

In Table 5, we compare the accuracy, run-time, and parameter size of various models. Initially, using the MLP-Mixer model on the EarVN1.0 dataset and training it from scratch yielded a test accuracy of 88.45%. This model performed better than MobileNetV2 but was behind VGG19 in terms of accuracy. In terms of parameter size, it was roughly eight times larger than MobileNetV2 and 0.12 times smaller han VGG19 (see Table 5). Similarly, by training from scratch using the ViT model, a test accuracy of 90.49% was achieved. This model was found to be comparable to VGG19 and MobileNetV2 in terms of efficacy and parameter size, respectively. In addition, the performance of hybrid models with MobileNetV2, MLP-Mixer, and ViT architectures was investigated. As seen in Table 5, test accuracy of 91.80%, 96.34% and 96.66% was obtained by using MLP-Mixer with ViT, MobileNetV2 with MLP-Mixer and MobileNetV2 with ViT respectively. These models were trained from scratch and end-to-end. In terms of both efficacy and parameter size, they surpass VGG19. Specifically, the suggested MobileNetV2 with ViT model could be preferred in terms of performance and parameter size; this model increased test accuracy by 5.54 percent compared to VGG19 and attained state-of-the-art results in the ear gender recognition field. In terms of parameter size, the magnitude of the suggested model was approximately 1/18 that of VGG19.

Table 6 illustrates the results of state-of-the-art models proposed for the ear gender recognition problem. Karasulu et al. [45] created a hybrid model by combining CNN and RNN architectures and achieved an accuracy of 85.16% on the test set. Nguyen-Quoc and Hoang [8] conducted experimental studies using popular CNN models with their default values and obtained the highest score of 91.1% with the VGG19 model. In terms of test accuracy, number of parameters, and training time, our proposed MobileNetV2 with ViT model provides a superior model architecture.

Table 6. Comparison of models

The hybrid model generated by integrating CNN, MLP-Mixer, and ViT has demonstrated superior performance in comparison to existing methods. The factors contributing to this performance improvement are as follows: (1) Complementary Feature Extraction: The combination of CNN, MLP Mixer, and ViT provides for complementary feature extraction capabilities. CNNs are renowned for their ability to identify local and spatial features in images, whereas ViT excels at identifying global context and long-distance dependencies. MLP Mixer complements these methods by enhancing the representation of features through token mixing. Using the strengths of each component, hybrid models produce a more comprehensive representation of the input data. (2) Hierarchical Information Processing: CNNs inherently capture hierarchical information through their stacked convolutional layers. MLP Mixers, on the other hand, introduce token mixing operations that enable the model to explicitly model relationships between different parts of the image. ViT uses self-attention mechanisms to capture global dependencies. By integrating these techniques, hybrid models are able to process both local and global data efficiently, resulting in enhanced performance. (3) Enhanced Representation Learning: The combination of different architectures allows for enhanced representation learning. CNNs, MLP Mixers, and ViT each have their own unique mechanisms for learning representations from data. By integrating these mechanisms, the hybrid models can capture a wider range of features and patterns, potentially leading to better discrimination and classification capabilities. (4) Adaptability to Data Characteristics: The hybrid models provide flexibility in adapting to different types of data characteristics. CNNs have been widely used for image-related tasks and are effective in capturing spatial information. MLP Mixers, with their token mixing operations, can handle various input sizes and effectively model relationships between tokens. ViT, with its attention mechanism, is capable of handling both image and sequence data. The combination of these models allows for a more versatile and adaptable approach to the task, accommodating different data characteristics that may arise. In summary, the proposed hybrid models leverage the strengths of CNN, MLP Mixer, and ViT to capture complementary features, process hierarchical information, enhance representation learning, and adapt to various data characteristics. These factors contribute to the observed performance improvements when compared to current methods.

5.png

Figure 5. Evaluating the models using the t-SNE method. The graphs represent the 0 (red): male and 1 (blue): female class

In Figure 5, the results of the t-Distributed Stochastic Neighbor Embedding (t-SNE) method are presented for evaluation of the models. t-SNE is a dimensionality reduction technique that is particularly effective for visualizing high-dimensional data sets and was first introduced by Van der Maaten and Hinton [46]. The t-SNE algorithm calculates a measure of similarity between pairings of samples in high-dimensional space and low-dimensional space; it then attempts to optimize these two similarity measures using a loss function. This method discovers a non-parametric mapping and provides an intuitive understanding of how the data is structured in high-dimensional space. The entanglement or disentanglement of features reveals the model’s performance. Examining Figure 5, it is observed that the results of the MobileNetV2 and ViT models are intertwined, making it challenging to separate the classes with a simple discriminative boundary. In our proposed MobileNetV2 with ViT model, the features obtained in the low-dimensional space are disentangled, and it is observed that the two classes can be easily separated. These results show that our proposed model is more successful for the ear gender recognition problem.