Video-Based Discrimination of Genuine and Counterfeit Facial Features Leveraging Cardiac Pulse Rhythm Signals in Access Control Systems

The accuracy of automated face recognition systems has seen substantial enhancements over recent decades. A myriad of advanced algorithms has been developed to overcome challenges associated with illumination, pose and expression variations, thereby enabling unconstrained face recognition across diverse applications [1]. However, despite these advancements, certain factors continue to impede the performance of access systems. These include issues pertaining to masked faces [2], printed faces, plastic surgery [3], make-up application [4], and spoofing faces [5]. For a face verification system to be effective in real-world scenarios, such as in securing access to restricted areas, a reliable system for distinguishing real and fake faces is essential.

Numerous approaches in the literature on fake face detection are based on the observation of texture properties in an image, comparing frames from a genuine person's face to spoofed face images.

In a recent paper, Kim et al. [2] proposed a method to distinguish a real face from a masked face using radiance measurements. This approach recovers the albedo from the reflectance of human facial skin and the reflectance of masked face materials (silicon, latex, or skinjell). Within a face verification system environment, Kim et al. simplified the computation of albedo from face reflectance, enabling the measurement of radiance for the forehead region under 850 and 685 nm illumination.

In another paper, Chen et al. [4] studied the differences between a make-up face and a non-make up face, and designed a method to detect the presence of make up on the face by extracting the features from the color, shape and texture characteristics for the ROI of the right eye, left eye and mouth cropped from the input face. In his experiments he used SVM to classify the make-up face and nonmake up face. On the print photo attack database [6] in the IJCB 2011, Chakka et al. [7] found the performance of six spoofing printed face detection algorithms on that database in a competition for measuring 2D face spoofing attacks. Määttä et al. [8] detected the spoofing face from a single image-based microtexture analysis, and found that for spoofing face detection, local phase quantization with a Gabor wavelet-based descriptor was less efficient than (LBPs) local binary patterns. Furthermore, they demonstrated that using three LBP descriptors of different configurations was more efficient than using LBPs with a single configuration.

Määttä et al. [9] also introduced a fusion system leveraging two descriptors from Gabor wavelets obtained from the local blocks of a face image, LBPs, and a histogram of oriented gradients. The histogram calculated for all the blocks was concatenated for each descriptor, producing three feature vectors. The classification was carried out using a linear SVM, with the final result derived from the fusion of the match scores of all three SVMs.

In another study aimed at detecting attacks from printed face photos, a fusion approach was employed which combined power spectrum and Local Binary Pattern (LBP) features, using a camera from an automated teller machine to compile the database [10]. These methods primarily relied on motion- and texture-based strategies, with facial movements leveraged to determine liveness.

Du et al. [11] undertook a project aimed at recognizing players' emotions during gameplay. This was achieved by measuring heart rate and detecting facial expressions in videos using continuous human emotion perception. In this work, a bidirectional long- and short-term memory (Bi-LSTM) network was deployed to learn the heart rate features, and a convolutional neural network (CNN) was trained to learn facial expression (FE) features. A self-organizing map back propagation (SOM-BP) was utilized to fuse these features and achieve an accurate match of the emotion.

Wang et al. [12] conducted a comprehensive review and comparative analysis of various methods for measuring heart rate from frontal face videos. The results of the survey indicate that using the heart rate signal extracted from facial skin as a source provides superior results, and that the independent component analysis (ICA) method outperforms others in signal extraction. Temko [13] introduced a novel algorithm, Wiener Filter and Phase Vocoder (WFPV), which employs a Wiener filter to reduce motion artifacts and a phase vocoder to refine heart rate estimates. Their heart rate estimation system was compared with existing algorithms on a database of 23 photoplethysmography (PPG) recordings.

Dautov et al. [14] implemented face detection using video plethysmography, with power spectral density (PSD) employed for heart rate monitoring. Nadrag et al. [15] utilized the Haar cascades method, initially proposed by Viola and Michel Jones, in combination with a fast Fourier transform (FFT) to identify the region of interest (ROI). Their paper presents a solution for measuring the heart rates of multiple individuals simultaneously, using object tracking to obtain a collection of face rectangles. The average color within the ROI is then used as the signal corresponding to the heart rate.

Zheng et al. [16] carried out heart rate predictions from facial videos with masks, using eye location as a reference point. Remote photoplethysmography (rPPG) was employed to extract signals from the face video, and a convolutional neural network (CNN) was utilized for heart rate detection.

Uppal et al. [17] conducted work on heart rate measurement using the brightness preserving bihistogram equalization (BBHE) technique. Their approach involves separating the captured image into red, blue, and green channels, with cheeks selected as the ROI. BBHE was applied to these regions, and heart rate measurement was performed using Principal Component Analysis (PCA).

Li et al. [18] proposed a framework that incorporates face tracking and the normalized least mean square adaptive filtering method. Discriminative response map fitting (DRMF) is employed to identify the ROI, and Welch's power spectral density estimation method is utilized for heart rate detection.

Gupta et al. [19] conducted heart rate monitoring using the Modeling and Bayesian Tracking (MOMBAT) method for detecting heart rate from face videos. Zheng et al. [20] detected heart rate using the symmetry substitution method, even in scenarios where facial details were lacking, such as during online classes. When the head is rotated approximately 30 to 40 degrees, causing partial loss of the ROI in the face, information from the left and right cheeks is symmetrically copied.

A comprehensive review was conducted by Rouast et al. [21] in which remote heart rate measurements were taken using low-cost RGB face videos. In their analysis, they emphasize that future remote PPG (rPPG) algorithms should focus on balancing the trade-off between the amount of processed data and the complexity of the algorithm.

Although remote photoplethysmography (rPPG) is capable of detecting heart rates from facial videos, the accuracy can be compromised by head movement. To address this challenge, Wang et al. [22] proposed an anti-motion interference method termed T-distributed Stochastic Neighbor Embedding (T-SNE) Based Signal Separation (TSS). In this approach, TSS initially decomposes the observed color traces into pulse-related vectors and noise vectors using the T-SNE algorithm. The vector with the most significant spectral peak is then selected as the pulse signal for heart rate detection.

Ibrahim et al. [23] introduced a method to address the obstacles in heart rate estimation from cameras that capture videos from long distances. In their research, facial landmarks are computed using a cascaded regression mechanism. The region of interest (ROI) is selected based on these facial landmarks, specifically where minimal nonrigid motion is identified. Temporal photoplethysmography (PPG) signals are extracted from the ROI, and environmental illumination signals are removed using an independent component analysis (ICA) filter. This PPG signal is further processed by applying a series of temporal filters to exclude frequencies outside the range of interest before determining the heart rate.

Zhao et al. [24] proposed a novel method to extract pulse signals from ROIs across multiple scales. Their research constructs a facial ROI pyramid of multiple scale levels. Blood volume pulse (BVP) signals are then extracted, and the final pulse signal is computed via signal fusion, to which a convex combination is applied.

Špetlík et al. [25] proposed a two-stem convolutional neural network for heart rate estimation. Heart rate estimation is achieved remotely by tracking the peripheral circulation of the blood, that is, through a non-contact reflective photoplethysmographic (NrPPG) HR detection. Their system operates on two components, the extractor and the heart rate estimator. The extractor's role is to process an input image and output a single number. This extractor is applied to a sequence of images to obtain a sequence of scalar outputs and NrPPG signals. These signals are then input into the heart rate estimator to determine the heart rate.

Related work on spoof face detection relies on complex configurations and texture analysis to yield satisfactory results. In contrast, our proposed approach for detecting the liveness of real faces is based on the analysis of cardiac pulse rhythm signals, employing video imaging and RGB analysis. We conducted experiments with 20 subjects, each recording two videos—one of their real faces and one of a printed face. This research can be summarized as follows:

1) A technique for a preprocessing method to detect the cardiac pulse signal from the green channel and red‒green channel in real face videos based on color RGB separation and signal enhancements is proposed.

2) Thresholds are set based on the difference between the max and min cardiac pulse signals.

3) We propose a liveness detection technique for real faces and spoofing detection for fake printed faces based on the thresholds from the cardiac pulse signals yielded from the green channel color of the ROI.

There are several proposed algorithms from existing works that measure the cardiac pulses from the face region only, with participants using video, that are non-contact [26, 27]. Some works use a thermal camera to monitor the change in the information in the thermal signal emitted from the vessels [28, 29]. In other works, a webcam camera was used to measure the cardiac pulses from the face using video imaging and separate the mixing channels; and FFT was used for the proper component after artifacts reduction from the separation channels [30, 31]. Another work used color magnification to amplify the color changes in facial skin [32].

There were artifact noise rejections, such as changes in the ambient lighting conditions and sudden motions of the participant [33]. We propose cardiac pulse signal detection using color RGB separation with artifact reduction, and we detect cardiac pulses from the green channel. FFT is used to obtain the frequency of the heart rate and then compares the results of cardiac pulses with a reference result from a commercial pulse oximeter, as shown in Figure 1.

1.png

Figure 1. Real face detection based on cardiac pulse signal from input video of face

The rest of this paper is organized as follows. Section 2 describes the work, experimental setup and the database used for this study. Section 3 describes the method for cardiac pulse signal detection and artifact reduction for the green component and red-green component, and then describes the method for real face detection. Section 4 presents the experimental results and the effectiveness of the proposed work in real face detection and spoofing printed face detection, and then discusses the results. Finally, Section 5 concludes the paper and discusses future recommendations.

Postprocessing of both the videos for real faces and printed faces was performed using demo software written in MATLAB version 2017b (The MathWorks, Inc.). Specifically, we used MATLAB's Signal Processing Toolbox to process and analyze the video data. This toolbox provides a comprehensive set of algorithms and tools for signal processing and data analysis. An overview of the main steps in this method to obtain the blood pulse from the real face video is outlined in Figure 3, and both signals from both videos are analyzed to set some thresholds to distinguish between the real face and fake face.

First, an automated face detector was used to detect the face within the video frames and localize the measurement region of interest (ROI) for every video frame. In this work, we used the face detector and nose detector based on the Viola and Jones algorithm for object detection [19], the algorithm returns the x- and y-coordinates along with the height and width that yields a box around the face for each face detected in the first fame for the input videos. From this box, we crop the center of the box by nose detection as an ROI for the subsequent calculations stated in Figure 2. The same coordinates of the selected region (ROI) are used for the entire sequence of frames in the input video.

To avoid face detection errors, if the face was not detected in the first frame, the face box coordinates from the next frame were used. If more than one face was detected, then the method takes the face box coordinates that were the closest to the box from the next frame. Some thresholds are set from the calculations for the ROI of all the frames, and our demonstration software shows a green box around the face with the label “Real Face” if the face is real in the input video, and shows a red box around the face with the label “Fake face” if the face is a spoof printed face on paper in the second input video.

3.1 Cardiac pulse signal detection and enhancement

A sum over all the pixel weights for every RGB channel in the ROI for each frame yields three components, r(t), g(t) and b(t) for red, green and blue, respectively, from all the video frames. The observed changes in the component plotting are based on the cardiovascular change events, as stated in Figure 4(a) for the g(t) component; always, the g(t) component is the best component for a cardiac pulse rate measurement [15].

The RGB sensors pick up other sources of fluctuation noise in the light with the cardiac pulse signals during video recordings of the participants due to artifacts such as changes in ambient lighting conditions and sudden motions of the participant, as plotted by the high amplitude changes of the red line in Figure 5(a). In this work, we used RGB sensors to detect cardiovascular changes in video recordings of participants. To process the RGB signals and reduce artifacts, we developed the following algorithm:

(1) Partition the green component of the acquired signal (20 sec video) into 40 partitions, as in Figure 5(a).

(2) Calculate the mean value of each partition, using the formula below:

$\bar{X}=\frac{\sum_{i=1}^n p g(t)_i}{n}$                   (1)

where, g(t)=Green Component, pg(t)=partition.

(3) Shift the samples of every partition to the mean level, using the equation:

$s g(t)=p g(t)-\bar{X}$                   (2)

where, sg(t)=Shifted partition.

(4) Repeat steps 2-3 above for all the partitions and then recombine the partitions to yield a cardiac pulse signal with a reduction in artifacts using the Eq. (3) and shown in Figure 5(b).

$\mathrm{G}(\mathrm{t})=[\mathrm{sg}(\mathrm{t}) 1 \mathrm{sg}(\mathrm{t}) 2 \mathrm{sg}(\mathrm{t}) 3 \ldots \ldots \ldots . \mathrm{sg}(\mathrm{t}) 60]$                   (3)

where, G(t)= recombined partitions.

(5) Use low-pass filter of (5 Hz) for the final enhancement as shown in Figure 5(c).

4.png

Figure 4. Real face video: (a) Green component and (b) Red‒green component with enhancements

5.png

Figure 5. (a) Partitioning the signal from the green component and then (b) Shifting all partitions to the mean level; (c) Final Enhancement with low-pass filter

Therefore, we have provided a detailed explanation of the steps involved in our algorithm. By partitioning the green component into 40 partitions and shifting the samples to the mean level, we were able to reduce the effects of artifacts and obtain a cardiac pulse signal with enhanced accuracy. We also applied a low-pass filter of 5 Hz for the final enhancement.

We believe that our algorithm is effective in reducing artifacts and enhancing the accuracy of the cardiac pulse signal. We have carefully selected the parameter values based on our analysis of the data.

In this work, there are four extraction components for each video, the red component, green component, blue components and (red + green) component. We use all the steps in Figure 5 of the enhancement of all the components (real face video and printed face video) for all the subjects, and then from this enhancement and artifact reduction, we can set accurate thresholds to distinguish between the real face and printed face.

3.2 Thresholds

The first threshold (T1) was set at a difference of 12 for the green components, while the second threshold (T2) was set at a difference of 18 for the red + green components. If the difference between the max and min weights for a vector exceeded these thresholds, we considered it to be a real face; otherwise, we classified it as a printed face. We also set an additional threshold of 100, which triggers a retest if the differences exceed this value, as shown in Table 1.

In addition, we also set a third threshold (T3) for the green components based on the number of pulses of any 4 sec with amplitudes greater than 12, with a minimum of 3 pulses required to classify a signal as a real face.

Table 1. Detection thresholds

*DGC: Difference between max and min in the green component.

*DRGC: Difference between max and min in the red + green component.

The specific threshold values used in our study were informed by previous research that investigated the cardiovascular events that occur in real human facial skin during the cardiac cycle [12]. We also considered factors such as the video capture equipment and lighting conditions. To validate our threshold setting method, we tested it on a sample of 20 participants with varying skin colors and ages, and compared the results to the ground truth obtained from manual inspection of the video data. The three final results were fused to the decision-maker for real face detection. For real face detection, the values of the results must be more than our thresholds. The three final results were fused to the decision-maker for real face detection.

The main use of this proposed work for liveness detection in face recognition systems is to access secure areas. Fake printed face detection must be rapidly improved to enhance facial biometric systems. We detected liveness in the real faces based on cardiac pulse rhythms. We extracted the cardiac pulse signal using the method detailed in Sections 2 and 3.

The strongest color change information in the ROI corresponding to the cardiac events was in the green component and red-green component. For some participants, the forehead is covered by a scarf or hair, and the chin region is covered by a beard. A small rectangular-shaped part of the nose and cheeks was selected as an ROI to reduce the computation time and complexity, and this ROI was used to avoid noise from eye blinking and from mouth motions. The experiments were performed at different times of day with different illumination ranges. We did not address how the proposed algorithm will work in a low lighting environment. Due to the small range of artifacts in our experiments from the motion and illumination changes, our proposed methodology can easily overcome these artifacts.

For further artifacts from rapid head movement, alternative artifact rejection could be used for improvements in future works [35, 36]. In addition, for this work, the video recording time was not short enough, future work still needs further reduce the video recording time to enable detection of real face liveness with less time.