Detecting Deepfakes: A Novel Framework Employing XceptionNet-Based Convolutional Neural Networks

The significance of social media in disseminating news has grown exponentially in recent years, leading to increased attention on online information operations. Advancements in computer vision methods have facilitated the creation of convincing fake graphics, which, coupled with the rapid dissemination of false news via social media, have exacerbated this issue. Contemporary data-driven approaches have simplified the generation of fabricated images from scratch, in stark contrast to the past, when image manipulation required extensive expertise in rendering and image manipulation tools. Consequently, deep fake videos or images have the potential to inflict unparalleled damage on political climates and individuals' private lives [1].

In recent years, technological advancements have enabled even non-experts to create deep fakes effortlessly [2]. These deep fakes involve digitally altering real people's faces in videos or photographs, often generated by generative adversarial networks (GANs), and can deceive audiences into believing that prominent figures are genuinely speaking when they are not [3]. As society becomes increasingly reliant on social media for critical information, the threat of misinformation from deep fakes intensifies. The need for software capable of detecting deep fakes—which can deceive the human eye—has become essential in light of the significant risks posed by false information. Given the rapid advancements in artificial intelligence (AI) in related domains [4, 5], it is evident that AI-driven solutions are necessary to counteract the detrimental effects of deep fake misuse [6].

In September 2020, prior to the U.S. elections, Microsoft introduced advanced technology for deep fake detection. To train or evaluate the deep fake detector, the Face Forensics++ dataset and the DFDC dataset were used by the Video Authenticator program. This program detects quiet or grayscale elements within the blending borders of deep fakes. However, most research in this area is still evolving to keep pace with the rapid development of deep fake detection and production, resulting in a largely concealed subject.

The term "Deepfake" is a portmanteau of "Deep Learning" and "Fake" and refers to content that has been technologically altered to appear photorealistic. This term was coined by an anonymous Reddit user in 2017 to describe the use of deep learning algorithms for replacing one person's face with another's in sexually explicit videos, producing highly convincing results [7]. Deep fake videos, created using deep learning algorithms, have gained significant attention recently. Rapid advancements in AI, ML, and DL have led to the development of novel methods and tools for manipulating multimedia content, paving the way for deepfake applications powered by deep learning [8].

Convolutional neural networks (CNNs) are used in various forms in all deep fake detection methods, as demonstrated by recent deep learning research [2, 9-12]. CNNs, which employ deep-learning, multi-layer classification neural network architectures, are capable of accurately identifying the visual attributes most relevant to a target category. Deep learning-based models often utilize CNNs due to their ability to detect areas of the frame with strong spatial and temporal connections. Extracting a face or facial features, such as the eyes or mouth, may help identify inconsistencies between real and deep-faked versions of an image. Several deep fake models employ CNNs [13, 14], albeit with differing implementations.

This study investigates the use of face recognition-extracted features within a classifier for the aforementioned data. As deep fake techniques alter a person's identity, they may modify their features. Current detection approaches typically rely on handcrafted low-level features, pre-trained object recognition embeddings, or CNNs, but rarely on face recognition networks [15]. A detailed examination of identity-separated extracted features is needed to improve deep fake detection.

The primary objective of this research is to implement a deep learning-based pre-trained CNN model, XceptionNet, using the DFDC dataset. A deep learning-based Xception Neural Network is proposed for detecting deep fake videos. XceptionNet is well-suited for image classification tasks for several reasons:

• Extreme depth: With 71 hidden layers, XceptionNet is a profoundly deep neural network, allowing the model to learn complex features and patterns from input data, which is especially beneficial for image recognition tasks.
• Separable convolutions: XceptionNet utilizes separable convolutions, a type of convolutional operation that separates spatial and channel-wise components of the convolution. This technique helps limit the number of model parameters, addressing the overfitting issue commonly encountered in deep learning.
• Better performance: XceptionNet has demonstrated superior performance on image classification tasks compared to many other state-of-the-art deep learning models.
• Lower computational cost: XceptionNet requires less computing power than other deep learning models due to the utilization of separable convolutions, which reduces the total number of training parameters.

The goal is to enhance the deep learning model's detection rate of deep fake videos. This research provides the following contributions:

• A comprehensive review of existing literature on deep fake detection, discussing current research tools, methodologies, and datasets while addressing specific research challenges.
• A proposal for a deep learning approach using Xception Neural Networks to detect fake videos.
• Improvement of the deep fake detection model by incorporating face recognition-extracted features, which can potentially identify inconsistencies between real and deep-faked versions of an image.

The remainder of the paper is organized as follows: Section 2 reviews deep fake generation methods and related deep fake detection techniques. Section 3 details the proposed deep learning-based approach for deep fake detection using Xception Neural Networks. Section 4 presents the experimental setup, including the dataset used, data preprocessing, and model evaluation. Section 5 discusses the experimental results, highlighting the performance of the proposed model. Finally, Section 6 concludes the paper and provides recommendations for future research.

3.1 Problem statement

Face recognition is one of the most challenging things in image processing due to the difficulties posed by developing systems for facial classification. Face recognition is the first concern to be solved. Face Detection issues may be attributed to the following two sources. Included are facial emotions and other facial characteristics. Facial expressions are one of the most powerful and immediate indicators of a person's personality since they reveal their emotions and intentions. Facial expressions of emotion, such as rage and happiness, may cause noticeable changes in a person's facial features. Quite a few people have specs, while some are sporting facial hair or scars from accidents or earlier injuries. Facial traits are the correct phrase for these features. New methods of detecting deep fakes have to be created because of their improving quality. Deep and shallow classifiers are the two most important classifiers for deep fake identification. Using DL models, fake photos and videos may be differentiated from authentic ones based on their distinctive characteristics. As an illustration, reflections in the eyes and other features may be lost. Similar differences in the teeth may be employed in the same manner.

3.2 Proposed methodology

To tackle the challenges above with deep fake, we will construct a deep learning-based Xception NN model to recognize deep fake movies using the Deepfake Detection Challenge Dataset in this research. Using facial landmark detection, we gather data relevant to different facial features from the videos. The raw video feed was assembled from the still images. The location of the eyes, nose, & lips has been determined using facial landmark detector. Eye blinks & other facial characteristics have also been identified using this data. There is some preprocessing work involved in supplying this information to the model. However, the images have been converted to their numerical representation via a preprocessing procedure. The first process involves resizing the input images and cropping them to focus on the faces. Now ensures that almost all pictures use the RGB channel. After finishing the preprocessing step, the training, validation, and testing sets were divided. The deep learning model undertakes feature extraction and train on the extracted features. To use this Xception Neural Network model, the classification step has determined whether or not a particular video is deep fake. Deep transfer learning is advantageous since it improves the model's interpretability and comprehension while reducing its computing cost. We will discuss findings on relatively quickly acquired Deepfake datasets with similar or better performance than existing DL-based algorithms.

1.png

Figure 1. Block diagram of the research methodology

Figure 1 depicts planned work's flowchart. In Figure 1, we see that this work starts with the data collection process, so we have used a deep fake dataset from Kaggle. The input dataset extracts the number of frames and then detects facial landmarks with the help of temporal features like eye, nose, lip, etc. After this, preprocessed the extracted images for image ROI face cropping, resizing, and converting images in RGB channels. The dataset has been divided into train, test, and Validation in the following proportions: 60%, 30%, and 20%. Here implements the deep learning using CNN-based pre-trained XceptonNet model for classification of deep fake. In conclusion, assess how well the suggested model performs in terms of its accuracy and loss. This complete method is being offered to determine if a video is fake or real.

3.2.1 Face landmark detection, and frame extraction

We have collected the 68 facial landmarks features from face images those landmarks features belong to eyes, nose, mouth, and edge of the face. We begin by splitting the input video into individual frames. The facial area is extracted from every image. Obtaining the face in X and Y coordinates for face recognition has various steps:

• Face Detection: Then, we use the Face detection technique from the lib package to identify any faces inside the picture.
• Face alignment: Once the face is detected, it is necessary to align the face using a landmark detection algorithm.
• Face encoding: After the face is aligned, it returns the set of features of landmark value such as the shape of face, eyes, nose, and mouth.
• Recognition area: The face is in image's recognition area, which is the area around it. Bounding boxes are often used to indicate the region of interest (ROI), which is the area around a detected face.

This landmark detector employs a group of regression trees to make estimations of face landmark coordinates using only the relative intensity of neighboring frames.

3.2.2 Eye blink detection

The vast majority of deep fakes by performers never blink nor blink abnormally. So, this nuance calls for a greater degree of skill to implement well. Next, we analyze the photographed person for any discernible blinking patterns.

The blink detector takes on facial landmarks as input, namely eye coordinates (points 37-46 in list of retrieved facial features). Soukupová and tech's formula for calculating the EAR (Eye Aspect Ratio) are used in the eye blink detector (2016). The six (x, y) coordinates that make up each eye are labeled from top left (p1) to bottom right (p6) in a clockwise fashion (p2, p3, p4, p5, p6). To calculate the EAR using this information, just use the formula in Step 1 below [40].

$E A R=\frac{\left\|p_2-p_6|1+1| p_3-p_5\right\|}{2|| p_1-p_4 \|}$     (1)

While an eye is open, the EAR is 1, but it drops to 0 when the eye is closed. An increase in blink rate is seen if the median EAR value of one eye goes below a threshold (EYE AR THRESH) for a consistent no. of frames (EYE AR CONSEC FRAMES). In this study, we have used values of 0.3 & 3 for EYE AR THRESH and EYE AR CONSEC FRAMES parameters, respectively.

3.2.3 Extraction of shape features

The facial landmark detector may be used to get the locations of the eyes, mouth, and nose. While face swapping is so common in deep fake films, adapting the shape of the eyes is a crucial aspect of the feature extraction process. During a real-world video, we found that a person's eye shape remained quite constant. When the film has been doctored countless times, it is no longer reliable. And similar to how most cases of facial warping occur close to the mouth, most cases of inconsistent facial features also occur there. Deepfake modification may also result in altered lip contours. As a result, we want to train our classifier using the differences in the shape of the facial features in various photos.

Coordinates (points 37-46) for the eye are provided into the eye form detector, which was determined using the facial landmarks. The furthest locations of the eyes are measured to calculate the left-right Euclidean distance (d1) and the right-left Euclidean distance (d2). Lip coordinates extracted from face landmarks are sent into the lip form detector (points 49-68). Euclidean distance between centers of the inner lips is used to determine their length (d3). As with the inner lip, the length of the outer lip is supposed to be determined by the inverse square of the distance between its coordinates as calculated using the Euclidean distance (d4).

The Nose Shape Detector gathers data about the nose from the facial landmarks (points 28-32). You can figure out how wide your nose is at its base by measuring the distance (in Euclidean units, d5) between your nostrils. You may approximate the nose's width by measuring the Euclidean distance (d6) between its extremities.

Hence, we can calculate the distance between inner & outer lip coordinates (d1, d2), width of nose (d3, d4), as well as width of eyes (d1, d2) (d5, d6).

3.2.4 Data preprocessing

Deep fake image classification requires pre-processing that improves the picture quality for further processing. Figure 1 depicts a flowchart of the preprocessing phase. Figure 1 depicts the three pre-processing steps: picture resizing, cropping the facial area of interest, and ensuring all photos are in the RGB color space.

Resize Image

All the images in the dataset were different sizes, and it would be impossible to get a reliable result from processing data of varying sizes. For the sake of processing, all photos were scaled down to 196×196. Downsampling, as well as upsampling techniques, were used to adjust the input image's resolution.

Region of Interest

ROI stands for "Region of Interest", It refers to a subset of data inside a larger collection that has been singled out for analysis. The idea of a profit from an investment is often used. For instance, data size may be determined by drawing its borders on an image or in a volume to analyze deep fake data. For face recognition, ROI is region within face's bounding box, which is determined by the face detection method.

Images Convert in RGB Channel

Red, green, and blue are the three colors that make up an RGB picture. The human visual system is made up of three-color receptors, as well as RGB channels are utilized in computer monitors and image scanners because they closely correspond to these.

3.2.5 Data splitting

Splitting data is a common practice in machine learning, and it's usually done to prevent overfitting. That's an instance of a machine learning model being faulty because it overfits its training data. In a machine learning model, the initial data is often divided into three distinct subsets labeled "train," "test," as well as "validation.". Here we split the dataset into in ratio of 60:20:20.

Training Set: A dataset is used to teach a model to identify anomalies and new patterns in the training data. 60 percent of data is in training set.

Validation Set: It's a different collection of data than the training set we use to check how well our model is doing while it's being trained. The validation set contains 20% of the dataset.

Test Set: The model is then tested on a different dataset when the training phase is complete. The training set is comprised of 20% of the whole dataset.

Table 2. Validation set size for training and testing, in terms of the number of photos

Above Table 2 shows the different number of images for each section of data i.e., training, testing, and validation. There are 2399 images for training data, 750 images for validation data, and 600 images for testing data given in this Table 2.

As was previously noted, the dataset was split as follows: 60% for training, 40% for validation, and 20% for testing, and the performance results on these ratios is given in Table 3. This is the conclusion we have reached, or so it seems:

Table 3. Performance results for different splitting ratio in terms of Training (60%), Validation (40%), and Testing (20%)

These variables would affect the outcomes if dataset was separated into a training (40%), validation (30%), and testing (30%) ratio. The Table 4 below shows the AUC score change and the validation accuracy.

Table 4. Performance results for different splitting ratio in terms of Training (40%), Validation (30%), and Testing (30%)

3.2.6 Classification using the DL model

Deep Learning [41, 42] is a well-known subfield within AI. It performs very well on a wide variety of data formats. When a DL model is fused with CNN, an image classification network may be produced. Figure 2 shows the skeleton of a typical CNN system. Digital pictures are stored in computers as collections of pixels that have common characteristics. Some groups of pixels in the picture may stand for edges, while others may depict shadows or other patterns. The use of convolution is one strategy for spotting such patterns. During processing, a matrix is used to describe the image's pixels. A "filter" matrix is increased by the matrix of picture pixels to identify the patterns. One may start convolution from the first pixel in the picture pixel matrix and choose pixels depending on the filter size for multiplication, where filter size determines the range of possible filter sizes. Convolution is performed for each pixel in the matrix until the whole picture has been processed. Numerous parameters are shared when convolving. A feature map is the layer-two output from a convolution. The pooling layer follows the activation layer in the CNN model. In order to prevent overfitting, this layer reduces size of final product, the feature map. Finally, on the topmost layer, all connections are made. A single vector representing the output of the current layer is "flattened" so that it may be utilized as an input by the subsequent layer. The feature maps obtained at each stage can be normalized using batch normalizations. Because of this, we employ dropouts to speed up the computation.

A DL strategy that uses CNN based on a pre-trained Xception Neural Network model is proposed in this article as a method for identifying false videos.

2.png

Figure 2. Classification using a fundamental CNN architecture

Pre-trained Xception architecture

The name Xception comes from the phrase "Extreme Inception." Google was the one that suggested using this approach. It utilizes the same parameters found in Inception V3 and comprises those parameters. The higher performance of Xception may be attributed to the effective use of the model's parameters and the system's expanded capacity. Under the inception framework, the output maps consist of cross-channel and spatial correlation mappings. The Xception architecture partitioned these mappings [43]. To extract features from the data using the network, 36 convolutional layers of the design were employed. These 36 levels are broken up into 14 different modules altogether. Linear residual connections encompass each module. Modules created first and last do not have any examples of this kind. The final FC tier will now consist of six courses instead of the previous seven. The proposed model contains some hyperparameters described below in Table 5 and a detailed function description.

The no. of epochs, batch size, learning rate, as well as loss function are all parameters that we set. Epochs are nothing more than the number of iterations, that is, 20, and each epoch lasts 20 years. By maintaining the batch size at its default value of 32, the no. of input samples sent to the network may be defined by the batch size some portions of the weights are adjusted when the neural network is being trained. These numbers are known as learning rates. This is a major CNN model hyperparameter, with values between 0.0 and 1.0. In our model, we used 0.0001 learning rates and investigated how different learning rates affected accuracy. Also, we used binary Crosse entropy loss function for classification models, which output a probability. A neural network's ability to accurately represent the training data is evaluated using something called a loss function, which is a function that contrasts the target as well as forecasted output values. During training, one of our goals is to reduce the deviation between the expected and target outputs.

Table 5. Setting of hyperparameters

The model's probabilities are compared to the actual class output, which may be either zero or one, using the binary cross-entropy method. Next, it assigns a value to each likelihood depending on how much it deviates from the expected value, and finally, it generates a score. It shows how near or distant we are from true value. Binary cross-entropy is described in this way:

$\prod_{c=1}^c y_c\left(x, w_c\right)^{t_c}\left(1-y_c\left(x, w_c\right)\right)^{\left(1-t_c\right)}$     (2)

3.png

Figure 3. Model summary of proposed Xception model

A description of suggested Xception model, which consists of various layers & parameters, is demonstrated in Figure 3. The total parameters of model are 97411,626, with trainable parameters of 97,357,098 & non-trainable parameters are 54,528, respectively.