Estimating the Smile by Evaluating the Spread of Lips

Smile is one of the important emotions that is essential in computer vision tasks. It is a good gesture presenting happiness, approval friendliness etc., Ochs et al. [1] propose a method to generate virtual smiles that conveys politeness, embarrassment, amusement and a combination of these. Machines understand any image with pixels and can estimate the smile by evaluating these pixel differences [2]. In real-time scenarios, few constraints affect smile detection. Those can resolve with proper image registration, feature representation and trained using a huge number of images [3]. An efficient deep Convolution neural network can combine feature selection and classification and can efficiently involve in smile prediction [4]. Since it involves two phases – feature extraction and classification, the accuracy of smile prediction depends on the accuracy of both the phases [5]. It is important to investigate the CNN for different kinds of transformations and distortions to understand its effectiveness in the realworld [6]. By extracting useful features [7], before applying CNN can improve the efficiency of detection. Instead of implementing CNN from scratch, using robust deep learning models that are used for face recognition, can attain the best smile detection results. Those systems not only improve accuracy but also reduces training time [8].

Not only deep neural networks but also few other approaches that can estimate the smile. Smile influences the mouth shape, by extracting these features using HOG and LBP Histograms, the smile can detect [9]. Edge oriented histogram [10] is one more feature descriptor that can detect the smile from lips. Yang and Zhen [11] propose a method of smile detection for mobile platforms using Gabor transforms. Compared to Gabor features, the pyramid histogram of oriented gradients is having a shorter vector length and a high smile recognition rate [12]. It is proved that the Fractional Fourier transform can also recognize smile better than Gabor features [13]. Instead of using one feature descriptor, using multiple descriptors, hierarchically can improve the smile detection accuracy [14]. By training the mouth and eye pair images using the KNN classifier, the smile on the face can detect [15]. The smile intensity can estimate by training the multiclass or regression models [16]. Akkoca and Gökmen [17] collected the images of smile and neutral and applied various approaches for smile detection. But this can’t be used in real-time because it can able to separate the smiling image from neutral but not from other emotions. Tang and Huang [18] proposes a method to characterize the smiles, by drawing the motion vectors of feature points of facial parts like eyes, mouth and cheeks that are derived from neutral and smiling faces. While smiling, the corners of the lip slightly curve up. Using Expansion of lips, the smile can detect and the contraction of eyebrows can estimate the frown. Tsai et al., and Royce et al. [19, 20] proposes Harris corner detection and FAST corner detection methods for identifying the smile by evaluating the curve up of lip corners.

From Figure 1, one can observe the expansion of lips due to smile. Hence by extracting the lips and estimating the distance it moves, one can find the smile. The colour of lips is always different from philtrum and chin colours. It is easier to segment the lips if a moustache and beard are present [21]. But this kind of segmentation doesn’t provide any information related to the spreading of lips. Haar cascade provides XML files for extraction of face parts like eyes, mouth, nose etc., But this kind of extraction also failed in providing expansion of lips, Facial landmarks can aid in providing information about all the face parts. This makes it useful in drowsiness detection [22], age estimation [23]. The expansion of lips can estimate using landmarks of the face.

In order to predict the smile face by evaluating the spread of lips, a threshold is defined. Threshold is the point between smile and no smile. From Figure 1 Threshold can define as (l = r) < x, can estimate as no smile and > x can predict as smile because spread of lips is equal at both the ends. But it is not true for all the cases. Figure 8 shows the one sided smiles having unequal l and r values. In some cases, even the two sided smiles failed to provide equal l and r values because of improper pose in the pictures. It is found that expansion of lip at any one side, can estimate as a smile. Hence, a parameter - Maximum of l and r is taken and derived a threshold in section 5.2. For improper posed images, the lips spread in one side more and the other side a little. In order to find the effect of combination of l and r, average of l and r is taken as a parameter and the derived the threshold in section 5.1. One more parameter Maxavgsum which is the combination of average and maximum is taken and estimated threshold is elucidated in section 5.3.

5.1 Average

The average is denoted with AANS for no smile case and AAS for smile case respectively. AANS or AAS = (l+r)/2 whose range is between 0 and 6. AANS and AAS can provide the information at a specific value i.e., at 0 or 0.5 etc., as illustrated in Table 6. But in order to find the spreading of lips, it is required to get sum of the values up to that specific value. Hence two parameters AAANS[k] $=\sum_{i=0}^{k} A A N S[i]$ for no smile case. The possible values of k are 0,0.5,1,-----,5.5,6. and AAAS[k] $=\sum_{i=0}^{k} A A  S[i]$  for smile case are defined and their respective values are illustrated in Table 6.

Figure 8. One sided smile images

Table 6. Threshold estimation for average

AAANS – Aggregate of average aligned no smile

AAAS – Aggregate of average aligned smile

9.png