Design and Development of Automated Student Attendance Framework in Fusion of CNN, HAAR, and ResNet

Attendance systems might be manual or automated. An event manager manually records attendance via a manual attendance method. Computerized attendance systems eliminate the tiresome task of manually tracking attendance. Attendance can be verified and recorded using authentication. Face recognition excels other biometric authentication systems regarding availability and discreet engagement [1]. Biometry identification is safe and accessible since it verifies identity using physiological and behavioral traits rather than knowledge or possession.

Knowing encrypted passwords allows an individual to replicate the identity of another user. Another individual can be verified through authorization-based authentication. Like credential-based and authorization-based authentication methods, biometric authentication systems, such as facial recognition, can present safety risks [2].

Forensic face recognition uses the physical characteristics of people. Profiling usually involves recording or photographing the appearance of the subject and using algorithms to extract jawline, distance between eyes, and nose curvature [3]. User impersonation is important to password-based authentication. They can also be verified by ownership-based authentication.

Research reveals that most facial recognition tools are susceptible to fraud attempts involving a perpetrator imitating an authorised individual to gain unauthorized use [4]. Face-resembling assaults are classified through three distinct categories in the study [5]. In level one, facial faking is performed with minimal or no before utilizing a mobile phone display or paper reproduction. The assailant must employ video footage of a face blinking and moving to prepare for Level B face spoofing. Level C: Attackers must use specialized equipment to create ultra-realistic 3D masks. For improving the privacy of a portable device’s student’s attendance system using face recognition, a liveness detection system that utilizes a Convolutional Neural Network (CNN) to analyses facial signals has been suggested [6].

Following are several notable advancements in the amount of face recognition-based student attendance systems:

•A convolutional neural network model for accurate face detection that uses transfer learning to build upon a model that has been trained

•Detection of continuous faces using an attendance system that uses live recognition of facial features

The subsequent content provides a concise overview of the following sections of the paper: Section 2 covers articles published in recent times on face detection and recognition. Section 3 discusses the proposed research methodology and includes data collection, image pre-processing, face detection, feature extraction and evaluation. Section 4 contains the suggested system’s experimental setup. Assessment of suggested system performance. Section 5 assesses the framework’s performance and compares it to other models. Section 6 contains information on ethical consideration and Section 7 concludes with further development.

Due to environmental constraints, acquired photos in real-world scenarios like classroom monitoring, surveillance, or mobile-based face identification sometimes have low resolution, poor illumination, or partial occlusion. These conditions impair the quality of facial characteristics, making successful detection and recognition difficult, particularly for deep learning models that rely on spatial resolution.

Traditional CNN designs, such as ResNet or HAAR-based detectors, are quite effective when fed high-quality frontal facial photos. However, their performance worsens when face input is distorted by size, noise, or pixel sparsity. These restrictions necessitated the development of a preprocessing module capable of recovering and improving facial detail from low-quality input images.

To solve this, we used a U-NET-based generator architecture, which has been extensively tested for reconstructing and enhancing spatial information from coarse, low-resolution images. U-NET, which was originally created for biological image segmentation, has demonstrated extraordinary accuracy in keeping fine-grained information while up-sampling. Recent researches [8, 12] demonstrate U-NET's utility in super-resolution facial augmentation, expression identification, and low-light recognition, hence supporting its application in facial recognition pipelines in limited situations.

In our approach, U-NET acts as a super-resolution enhancer, upscaling images from 64 × 64 to 256 × 256. This enables HAAR and ResNet-based recognition layers to operate on feature-rich facial inputs. This multi-model synergy improves overall system robustness and allows for consistent recognition across a variety of classroom circumstances.

1.png

Figure 1. Proposed architecture of face detection and face recognition framework

The fundamental principles for the development and utilization of such systems include the provision of comprehensive data protection, the provision of informed consent, and the implementation of frequent technical updates to address potential biases. This approach necessitates the registration of unique IDs, the generation of databases, the detection of faces, the training of features, the recognition of faces, and the development of a graphical user interface. The architecture and processes of the system are delineated in this section. Figure 1 illustrates that the system comprises two algorithms: face detection and recognition.

3.1 Data collection

Our research is significantly influenced by the data obtained from images. The dataset consists of 1120 student face photographs, each captured from three distinct angles (front, left, and right). The pictures of students’ faces have been saved in the format student roll number.jpg and labelled with student name and student roll number. A numerical value is assigned to each angle orientation in the file name. The folder name illustration is depicted in Figure 2, and the following figures and angles are as follows.

2.png

Figure 2. Sample image collection of student facial from three different angles

3.2 Image preprocessing

Super-resolution in subject photos is often needed and complicated. Low-resolution photos require super-resolution to provide more precise results. Deep learning face recognition algorithms have been studied for image super-resolution; however low-resolution face detection is difficult. To address this issue, researchers proposed a generator architecture similar to the U-NET [12]. Our generator architecture uses the nearest neighbor method to scale a 64 × 64 input image to 256 × 256 mentioned in Figure 3. Our model needs this architecture to produce high-quality photos.

3.jpg

Figure 3. Detailed architecture of U-NET style image generator

The researchers employed image enhancement and scaling. Initially, the contrast of the image was enhanced by histogram equalization. We utilized feature scaling to scale our datasets to 128×128 dimensions using RGB images. We normalized the dataset and scaled the photos as part of feature scaling. The CNN is terminated more rapidly by data between 0 and 1 than data between 0 and 255. This mitigates disparities in illumination.

3.3 Face detection

Oriented gradient histograms are “Feature Descriptors” in feature engineering, which extract structure-based information from images. This procedure divides the image by calculating histograms and defining local sections. Two matrices are used to calculate gradients: magnitude changes in relation to the x and y axes. Calculating the gradient change in the x-direction is illustrated in Figure 3.

Pixels can represent the nasal component as a matrix. The highlighted pixel is updated by subtracting the correct value from the left side due to a horizontal change. All matrices change in both directions. Pythagoras orients pixels. Next, we generate a histogram from pixel continuous value categories. Despite being older than other detection systems, HAAR remains relevant due to its advanced computational capabilities in evaluating static and moving photos. A simpler matrix is obtained via convolution of photos with a kernel of varied scales. Non-facial data points are “no” and facial data points “yes”. Nose, eye, and cheek detection require these levels.

3.4 Feature extraction

Constructing deep convolution neural networks with vast datasets is time-consuming and expensive. The computational feasibility and time are reduced by reusing model weights from pre-trained models constructed as benchmarks on large datasets such as ImageNet [13]. The model may employ a number vector to characterize a feature in Figure 4 by analyzing the characteristics of the retrieved image. Although there may be superior image recognition models, we will select the most effective classification model due to the following:

4.jpg

Figure 4. The rate of change of specimen gradients in the X direction for a particular pixel

3.4.1 VGG16

On the ImageNet dataset comprising 14 million images and 1,000 classifications, one of the numerous prominent deep convolutional neural networks employed for image identification achieved a 92.7%. 16 of the 41 layers in the network can be trained with weights, including 13 convolutional and 3 fully connected, and can accommodate (224, 224, 3) images. The VGG-16 design with 138,357,544 parameters is illustrated in Figure 5.

5.png

Figure 5. Detailed architecture of VGG16

3.4.2 Face recognition using transfer learning with ResNet

Deep residual network ResNet50 is a widely used architecture for the training of highly complex neural networks. Residual blocks facilitate the training of deep networks and mitigate the vanishing gradient issue. The network employs convolutional layers with skip connections in each block to facilitate efficient training with a high level of depth.

$ Output =F(x)+x$            (1)

$G A P=\frac{1}{H * W} \sum_{i=1}^H \sum_{j=1}^W$ FeatureMap $(i, j)$            (2)

$\begin{gathered} Output=Activation (Weights * Features + Biases )\end{gathered}$            (3)

This summing procedure (Eq. (2)) is indispensable for residual learning. The model simplifies the spatial dimensions of the previous layer into a vector representation by calculating the average of each feature map using Global Average Pooling (GAP) (Eq. (3)). Lastly, the fully connected layer linearly modifies features from antecedent layers using learning biases and weights (Eq. (3)). This results in forecasts or categorization. Our ResNet50 model employed weights trained on ’ImageNet’ to utilize the collected features. The model was subsequently refined on our dataset using a fully connected layer with three neurons and a global average pooling layer.

3.4.3 Face recognition using transfer learning with inception v3

The Inception V3 TL model is renowned for its design, which incorporates numerous ’Inception’ elements. These modules excel in photo categorization due to their simultaneous processing at differing sizes. As illustrated in Eq. (4), Inception modules can perform numerous concurrent convolutions. The kernel diameters of its convolutional layers are variable. The network may acquire features at varying scales by incorporating convolutional processes with varying kernel sizes and pooling. The GAP layer averages each feature map, similar to ResNet50, to convert the spatial output from the antecedent layer to a vector (Eq. (4)).

$\begin{gathered}O / P=Cancat(Conv(1), Conv(3), MaxPool(3 * 3))\end{gathered}$          (4)

The three-neuron final layer (Eq. (7)) linearly transforms the learned biases and weights to generate predictions or classifications. Ultimately, this is done.

3.4.4 MobileNet

MobileNet, a lightweight TL model [14], accelerates the inference process for mobile and embedded devices [15]. It is characterized by its diminutive model size and low computational expense. The convolution of each channel determines depth-wise convolution i and filter k using distinct filters. The input, depth-wise filter, and output are defined in Eq. (5), with the input being Ii,j and the depth-wise filter being Ki,j. The pointwise convolution applies (1 * 1) convolutions across the depth of the intermediate feature maps from the depthwise convolution. The intermediate feature map I¯i,j,c, weights Wc,k, bias bk, and output feature map are illustrated in Eq. (10). The GAP layer generates a vector representation by aggregating the values of all feature maps across all spatial dimensions, as described in Eq. (6).

$l_{i, i, k}=l_{i, i} * K_i, k$             (5)

Lastly, Eq. (6) illustrates the linear transformation of GAP vector representations by the fully connected layer, which employs weights and biases. Classification is achieved through modification.

$O_{i, j, k}=\sum_{c=1}^c I_{i, j, c} * W_{c, k}+b_k$         (6)

3.5 Evaluation

The authors evaluated our photo classification models by utilizing the F1 score, precision, macro average accuracy, and recall. The all-class mean in Eq. (7), Macro Average Accuracy, determines accuracy. It is essential for the assessment of model efficacy. Macro average recall across all classes is quantified by Eq. (8). The model’s ability to accurately categories instances of each class is assessed. The Macro Average Precision determines the class-wide precision (Eq. (9)). The model’s capacity to forecast class outcomes is assessed. The average F1 score across all courses, including precision and recall, is determined by the Macro Average F1 Score (Eq. (10)). It evaluates the performance of the model fairly. Here, the number of classes is denoted by N. The negatives and true positives for class i are represented by TNi and TPi. In the class i, false positives are defined as FPi and false negatives as FNi.

macro_acc $=\frac{1}{N} \sum_{i=1}^N \frac{T P_i+T N_i}{T P_i+T N_i+F P_i+F N_i}$           (7)

macro_rcall $=\frac{1}{N} \sum_{i=1}^N \frac{T P_i}{T P_i+F N_i}$            (8)

macro_precision $=\frac{1}{N} \sum_{i=1}^N \frac{T P_i}{T P_i+F P_i}$              (9)

macro_f1_core $=\frac{1}{N} \sum_{i=1}^N \frac{2 . T P_i}{2 . T P_i+F P_i+F N_i}$           (10)

The suggested model is tested on comparable datasets in this part. Since CNN forms the basis of the proposed method, we begin by conducting extensive CNN trials to establish a performance standard. Try different VGG-16 fine-tuning parameters. Finally, we compare the proposed model to pretrained ones. The accuracy of a Convolutional Neural Network with multiple layers, a 3x3 kernel, and 2x2 MaxPooling is shown in Table 1 for the Testing Group algorithm. The datasets evaluated include KDEF and JAFFE, with input sizes ranging from 360 × 360 to 48 × 48. Further, to ensure that our suggested attendance paradigm is reliable across multiple populations, we tested its recognition accuracy on two large-scale, ethnically diverse datasets: CelebA and Racial Faces in the Wild (RFW). The CelebA dataset contains over 200,000 celebrity photos annotated with variables such as gender, age, and ethnicity, offering a large testing ground for assessing model robustness under different expressions, lighting, and position situations. The RFW dataset, on the other hand, is specifically designed to assess face recognition bias across various racial groups, including Caucasian, Asian, Indian, and African. Its major goal is to assess the fairness and demographic performance disparities in facial recognition systems. The test magnitude was determined by randomly selecting 10% of the available data. We show the optimal test set accuracies for a configuration for all 50 iterations. The table implies that larger input photographs maximize accuracy for both datasets. Using the same dataset, KDEF accuracy is 74.81% for 360 × 360 input images and 65.78% for 48 × 48 input images. Larger photos include more data, so categorization methods work better. The most extensive input size (360 × 360) did not achieve optimum accuracy. The most accurate image resolutions for both datasets were 128 × 128. The model fits the photo input size, but greater input sizes require a larger model and more data.

Table 1. Accuracy of two-layer conventional convolutional neural networks on the KDEF, JAFFE, CelebA, and RFW databases for images of varying input sizes

To test the generalizability of our proposed paradigm across demographically different groups, we expanded our studies to incorporate the CelebA and Racial Faces in the Wild datasets. CelebA and RFW provide more range in pose, illumination, age, gender, and racial dispersion than the JAFFE and KDEF datasets, which are limited in ethnic variance and controlled conditions, respectively. As expected, these datasets had slightly poorer recognition accuracy due to greater intra-class variability and real-world noise.

The model recognized faces with a maximal accuracy of 76.31% on CelebA and 72.44% on RFW at an input resolution of 128×128 pixels, indicating robustness in uncontrolled, diverse situations.  Lower accuracy on the RFW dataset can be attributed to the challenges posed by racial imbalance and dataset complexity, which are consistent with current studies.  Nonetheless, the performance is still competitive, demonstrating the adaptability of our U-NET + HAAR + ResNet infrastructure across several ethnic groups.  These findings support the viability of our technique for use in real-world attendance systems where demographic fairness is crucial.

Various fine-tuning methods were implemented to understand their function in the proposed TL-based system. Table 2 illustrates the accuracy of the VGG-16 model on the KDEF and JAFFE datasets with a 10% random sample, subject to various fine-tuning options. In any fine-tuning mode, we trained dense layers for 10 iterations. Subsequently, we added Conv blocks to the pre-trained foundation for 40 iterations, resulting in 50 iterations in the model. The results of the complete model’s training from inception (randomly initialized) for 50 iterations are presented to evaluate the efficacy of the TL based technique. The table demonstrates that the pre-trained model’s final block (Block 5) yields significantly superior results when fine-tuned than when fine-tuned solely with dense layers. While fine-tuning, it is essential to contemplate the entire VGG-16 Base, not just Block 5. The highest accuracy is 96.78% in the KDEF dataset and 100% in the JAFFE dataset. In contrast to the TL-based fine-tuning mode, the KDEF and the JAFFE datasets exhibit extremely low accuracies when the complete model is trained from scratch: 23.35% and 37.82%, respectively. The proposed TL approach was validated in the findings table, and a pre-trained DCNN model component was refined.

A remarkable result is that a deeper model outperformed other, as indicated in Table 2. For example, ResNet-152 achieved better accuracy than ResNet-18 on the KDEF test set for the 10-Fold CV case, with 97.12% and 94.17%, respectively. DenseNet-161 surpassed the pre-trained DCNN models on both datasets. In ten 10-fold CV iterations, the model accurately identified 1122 KDEF and 1 JAFFE samples. The presented architecture accurately identified students by employing face detection and recognition techniques (refer to Figure 6).

Table 2. Test set accuracy comparison with pre-trained deep CNN models on KDEF and JAFFE datasets

Table 3. Comparison of suggested framework results with different ML-DL models

Table 4. Comparison of suggested framework results with different ML-DL models

The proposed framework classified pupils more accurately than other machine learning and transfer learning models (97.81%), as demonstrated in Table 3 and Table 4. In terms of accuracy, recall, and F1-score, the framework that was proposed outperformed its competitors.

Numerous investigators have invented attendance marking automation systems, but only some have been in the literature survey.

Table 5 compares inference time and computational complexity (measured in GFLOPs) for several machine learning and deep learning models. Traditional machine learning models like Random Forest, SVM, and XGBoost have low FLOPs (<0.02) and fast inference times (<7 ms), making them highly computationally efficient. In comparison, deep learning models have higher learning capabilities despite being much more demanding in terms of FLOPs. Among these, MobileNetV2 strikes a good compromise between speed (12 ms) and complexity (300 GFLOPs), whereas the suggested U-NET + HAAR + ResNet50 framework, while heavier (4800 GFLOPs), retains a practical inference time of 35 ms while improving performance.

Table 5. Comparative performance of traditional machine learning and deep learning models based on inference time (per image), and computational complexity (FLOPs)

The aggregate detriment of all models in relation to the recommended framework 7(a) is illustrated in Figure 7. This statistic considers two parameters: train loss and values loss. It has been observed that SVM and VGG-19 experienced significant data loss throughout the training process.

Table 6. Suggested framework results compared to recent suggested framework results

Table 6 compares the proposed framework to other contemporary frameworks based on macro accuracy. A macro accuracy of 97.81 percent was attained by the framework that was recommended, which achieved greater performance in comparison to the most recent framework that is considered to be state-of-the-art in the field of face detection and recognition.

6.png

Figure 6. Results of proposed U-NET + HAAR +CNN + ResNet 50 face recognition framework

7a.png

7b.png

7c.png

7d.png

7e.png

7f.png

Figure 7. Training and test loss of all the models compared with the proposed framework

The following are two essential elements of the suggested system that contribute to an increase in the precision of attendance recordings. An automatic attendance authentication system that is based on face recognition is the best choice because it is straightforward and requires only a small number of acquisition appliances. In the second step, it is necessary to train models with limited datasets and resources and to include advanced mobile devices that have limited RAM and storage space in highly mobile environments. When it comes to machine learning systems, selecting parameters is quite necessary. The hyperparameters were chosen after many pipeline training cycles were completed. This set of hyperparameters includes the number of neurons, dense layers, and optimization learning parameters. By optimizing the parameters of the suggested model for each dataset, it is possible to improve the performance of the approach that has been proposed.

The proposed research aims to develop a method for automated student attendance systems using face recognition. Consequently, the proposed system implemented the technique to automatically identify and document the attendance of all pupils in a specific classroom. The mask intended to conceal most of the mouth, nostrils, and cheeks complicates the process of identifying facial features. However, the U-NET and HAAR cascade enables the model to extract more information from fewer attributes. The ResNet assistance in parameter setting and neighbors selection facilitates the identification of pupils with high degrees of certainty. The proposed method can be deployed to find the footfall in numerous public locations, such as supermarkets, offices, businesses, train stations, malls, colleges, and airports. The authors aspire to eventually be able to autonomously mark attendance in Excel and identify individuals’ faces in images containing multiple faces.