Ululate: A Non-Intrusive, Wearable Tongue Gesture Detection System for Human-Computer Interaction

Human-computer interaction is the study of the interface between humans and computers. The HCI system integrates various fields of study, such as computer science, human factors, psychology, and ergonomics. Furthermore, HCI deals with questions of human perception, intelligence, decision-making, and interactive techniques of visualization, so it centers mainly on supervised methods.

With the improvement and exploitation of the sensor technology, it is important to improve micro-interaction with wearable devices because its applications may affect the user experience, whether it is enabling computers to be accessed by individuals with quadriplegia or just being used on a regular basis.

Due to modern technology, a growing interest in human-computer interaction has been observed in the last few years. HCI is attracting significant interest due to its importance in improving the user experience in a wide range of domains. HCI applications can vary from medical applications to wearable devices, security, and automatic vehicles. In medical applications, using an advanced HCI system as an input device is vital for a wide range of users, including quadriplegic patients and those with mutism making it easier for them to engage with smart devices. Furthermore, stress detection, where HCI systems show an improvement in detecting human stress levels using ECG signals, may appear to be a challenging task. Still, with the development of wearable devices and the improvement in sensor technology, it is now possible to detect stress levels using ECG signals where the sensors are located on the wrist that can be detected, identified, and diagnosed. In addition to measuring stress, those systems can also detect fatigue of construction workers, drivers and measure glucose levels in the blood of patients with Diabetes [1-12]. Understanding human-computer interaction is critical for creating positive user experiences. Mental health specialists from all disciplines are increasingly creating and implementing Internet-based therapies for persons suffering from a variety of mental illnesses. Technology-enabled therapies for a variety of mental health conditions have several therapeutic and economic benefits. Despite that, the influence of HCI and associated design aspects on patient safety, efficacy, and treatment adherence for computer users who engage in e-mental health therapies still need to be discovered [13].

While most electronic courseware enables users to progress at their own rate, courseware designers frequently make two assumptions. Initially, all users are presumed to be able to synthesize graphical material with their existing experience and knowledge to facilitate learning without considering diverse cognitive types. Recognizing student characteristics becomes crucial in broadening access to electronic information. Secondly, learning is taken for granted rather than validated. Therefore, HCI systems employ data analysis techniques to differentiate between what individuals comprehend and what they do not. E-Learning websites could utilize the HCI system to improve the learning experience by collecting and analyzing user data when they use learning websites [14-17]. The security research community acknowledges user behavior as a significant factor in various security vulnerabilities. Characterizing people as the "weakest link in the security chain" has gained popularity. Designers must identify and tackle the root causes of undesired user behavior to develop effective security systems. Researchers can draw upon knowledge and techniques from human-computer interaction to proactively address and resolve these issues [18-22].

Hands are required in today’s interface, which causes limitations when running or walking. Voice- based HCI might solve these problems, but it creates other issues. For example, using voice- based HCI in crowded areas is inappropriate for privacy reasons. Furthermore, voice-based HCI is inefficient in noisy places due to cross-talk and background noise. Autonomous systems powered by artificial intelligence (AI) are progressively infiltrating people’s daily lives and work. Autonomous cars are a prominent category. In the last few years, researchers have shown much interest in researching smart cars. Integrating visual human- computer interactions (VHCIs) into intelligent vehicle systems is critical for closing the gap between autonomous and man-machine control systems. Concerned about autonomous systems’ safety, researchers have emphasized that HCI systems for automatic cars must prioritize human factors in design for this new technology class. They suggest that society is undergoing considerable transformation due to the emergence of AI-based autonomous systems that differ from traditional automation. There needs to be more clarity between automation and AI-based autonomy, which may lead to unrealistic expectations and the misuse of technology [21-23].

The daily maintenance and repair of a specific type of equipment are often hampered by long fault diagnosis times and untimely repairs. By using HCI systems, analog electronic circuits and equipment can be diagnosed and identified for faults, preventing accidents from occurring [24, 25].

The main principle of creating an HCI system is to create a functioning system that considers various HCI system design concerns and solves them in a way that does not impair system efficiency while improving user experience. We may divide HCI system design limitations into usability and user experience, with usability constraints being any component that influences system function, such as efficiency, safety, and utility. On the other hand, user experience limitations relate to how users interact with the HCI system, so increasing system complexity might negatively impact the user experience.

The interest in HCI is focused on solving issues in various design aspects. The design of an HCI system must consider multiple factors that improve the user experience, such as privacy, aesthetics, hygiene, and system complexity.

Bulky gadgets on the hand or head are unsightly and obstruct daily activities. In addition, obtrusive HCI devices can be unattractive, especially in public places. As a result, while building an HCI system, we must consider numerous aesthetic factors like device size, weight, positioning, and employing origin to control the device.

Voice assistants are digital helpers that can understand and respond to human speech using computer-generated voices. The most popular voice assistants are Apple’s Siri, Amazon’s Alexa, Microsoft’s Cortana, and Google’s Assistant, which can be found on smartphones or smart speakers. Users can speak to their assistants to ask questions, control home automation devices and media playback, and manage tasks like email, to-do lists, and calendars.

Voice assistants are handy tools but also have significant security risks. One of the most critical challenges is ensuring the security of the personal information stored on these devices. Since voice-activated devices can read calendar events, emails, and other personal information, anyone with access to them can quickly access this sensitive data. Additionally, voice assistants are vulnerable to different types of attacks. Researchers have shown that voice assistants can respond to inaudible commands delivered at ultrasonic frequencies. Attackers can use this technique to gain control of the device, and this type of attack could be broadcast over the airwaves in the future [26, 27]. Therefore, voice-based HCI in crowded areas is not recommended for privacy considerations. Furthermore, voice-based HCI is inefficient in loud environments due to cross-talk and background noise.

Human-Computer Interaction has been rapidly growing and has introduced a new area of research known as tongue-based HCI systems. These systems offer a potential input method for individuals who have limited or no mobility in their limbs. However, creating and deploying such systems has significant challenges that must be carefully addressed.

Choosing suitable sensors and location is crucial for accurately detecting tongue gestures in HCI systems. Many proposed input devices require invasive procedures like tongue piercing or multiple electrodes, which could be unsuitable for public use due to hygiene and aesthetic concerns.

Moreover, deploying in-mouth devices, such as magnetic implants, poses significant concerns regarding oral hygiene. In this context, a wearable sensor device on the Genioglossus muscle has been devised to address these issues. This innovative approach eliminates the need for regular cleaning and minimizes the requirement for additional assistance during implantation.

In response to these limitations, we have developed "Ululate," an advanced hands-free HCI system designed as an alternative input method. The name "Ululate" draws inspiration from the Arabic term "Ululation," which signifies a long, wavering vocal sound produced by rapid tongue movements. This term is often associated with celebratory expressions of joy in African and Middle Eastern cultures, typically observed during communal ritual events, such as weddings. The name "Ululate" conveys the resemblance of tongue movement without implying the accompanying vocal sound.

In this section, we briefly introduced most of the aspects of HCI design. The following sections are organized as Literature Review, Methodology, Results and Discussion, and Conclusions.

One of the primary goals when developing the Ululate system was to make it acceptable for everyday use. As a result, when creating Ululate, we examined each approach in terms of non-intrusive, unobtrusive, non-invasive, and other design aspects.

Selecting the best sensing unit is one of the most significant aspects since it imposes constraints on the system. When selecting sensors for the Ululate, we investigated many sensors and methodologies for tongue detection, including textile pressure, X-band Doppler, and EEG/EMG sensors. Furthermore, due to its size, weight, accuracy, and low cost, we determined that an accelerometer sensor was the best match for the system.

Examining the optimum position for the sensing unit is one of the most critical and challenging aspects of the design process since it is a trade-off between what the user prefers and what is optimal for the sensing position. As a result, the sensors were placed on the Genioglossus muscle for maximum accuracy.

The gesture is a form of body language communication that may be described with or without spoken words. The primary goal of gesture interaction in HCI is to develop systems that can recognize specific human gestures and utilize them to communicate information or control devices [35, 36]. As a result, while choosing the best tongue gestures, we assessed numerous factors, including the intricacy of the gestures, whether they were short and direct, and if they were better if fewer organs were employed. Furthermore, we employed several tongue movements gestures to investigate the concepts of tongue movement as an input device and determine which gestures would be best suitable for consumers. As a result, four distinct tongue gestures are proposed.

As a result, a sensing device equipped with an accelerometer was designed in order to construct an HCI system based on tongue gesture recognition. The sensor unit was then placed on one test participant’s lower jaw on the Genioglossus muscle, and we performed several tongue movements gestures to evaluate the concepts of tongue movement as an input device and discover which gestures would be most appropriate for users. As a consequence, four distinct tongue motions are proposed. Finally, measurements are collected, evaluated, and fed into supervised machine-learning algorithms to classify gestures. Furthermore, we computed and compared system accuracy with four different machine-learning classifiers and accuracy matrices.

3.1 Ululate system design

Ululate system comprises a compact sensing unit connected via USB to the computer. The sensing unit includes a 29×58 mm ESP32 micro controller and a 3×5 mm ADXL345 accelerometer sensor positioned below the lower jaw on the Genioglossus muscle, as shown in Figure 1. The accelerometer data were collected and saved on the computer.

As the accelerometer chip in Figure 2 has a modest profile (less than 1 cm²), it could be effortlessly included in a scarf or shirt collar since the proposed solution must be undetectable.

1.png

Figure 1. Sensing unit positioned on Genioglossus muscle

2.png

Figure 2. Accelerometer chip

3.2 Gesture selection

To design an intuitive HCI system, it’s essential to understand users’ preferences for these gestures, especially the if the study aims to create an everyday HCI system. Chen et al. [28] offered taxonomies to delineate the formation and application of micro-mouth gestures. They introduced a practical collection of 20 mouth micro-gestures, selected based on user preference, suitable for performing tasks in common software applications [35-37].

As the tongue-based HCI system is an unfamiliar mode of interaction, taking into consideration user preferences, the selected gestures should be short, direct, and batter if fewer organs are used. Furthermore, natural mouth and tongue gestures cannot be considered since their resemblance to everyday behavior may make them poor selections for gestures. Therefore, we propose four different gestures. The selected gestures are suitable for various users under different conditions since they are not complex and need just one moving organ, and because tongue movements work while the mouth is closed, they are more user-friendly.

Consequently, four tongue gestures were chosen: the first two are fast and slow vertical Ululate. During these two gestures, the tongue travels up and down, contacting the upper and lower jaws while the mouth is closed. The other two gestures are fast and slow Ululate, wherein these two gestures, the tongue moves left and right, contacting the inner side of the gums, as shown in Figure 3. Notice that the gestures do not require movement of the jaws or mouth opening, which is essential to make the proposed HCI system unobtrusive.

3.png

Figure 3. Ululate gestures

3.3 Conducting experiments and feature selection

The experiment used data from one participant. The sensing unit was attached to the participant's Genioglossus muscle below the lower jaw using double-sided tape.

The sensing device was connected to the computer through a USB connection, and the experiment was conducted sitting, with participants requested to execute the specified gesture followed by no tongue movement.

In the experiment, a single test subject, a 23-year-old female, participated in the test phase. Prior to commencing the data collection, all gestures were thoroughly explained to the subject through both visual aids and in-person descriptions. The test subject was then instructed to practice the selected gestures during the initial phase of data collection.

Subsequently, the experimental process commenced, and each trial lasted for 80 seconds. Throughout the experiment, the sensor recorded 100 data points per second, resulting in approximately 8000 data records for each class of gestures.

In the feature selection process, we utilized raw coordinate data obtained from the sensor and further extracted additional properties, including mean, standard deviation, minimum, and maximum values. A time frame of ten consecutive data points was employed to capture temporal information to derive new characteristics. The resulting feature set comprises 12 distinct features with a corresponding label.

The chosen features are considered relevant for the time series classification process for several reasons. Firstly, extracting statistical properties such as mean, standard deviation, minimum, and maximum allows us to capture essential information about the distribution and variability of tongue movements, which could indicate different gestures. Secondly, the incorporation of temporal information by using a time frame of ten data points enables the model to consider the sequential patterns and dynamics in the tongue gestures, which can be crucial for accurate classification.

By including these specific features in our dataset, we aim to enhance the discriminatory power of the machine learning and deep learning models when differentiating between tongue-based HCI gestures. The relevance of these features lies in their ability to encapsulate essential characteristics of tongue movements, contributing to a more robust and accurate classification process.

3.4 Data classification

Classification is an essential step in putting the HCI Ululate system into action. Classification determines a new data class based on a training data set with known class membership. In this study, offline classification was done using four supervised machine learning algorithms, i.e., SVM, KNN, random forest, and Decision tree. Due to their proven effectiveness in previous studies, these machine-learning models have shown remarkable classification accuracy when utilized for the classification of tongue-based HCI systems. Next, we compared each algorithm's accuracy using five different accuracy matrices.

KNN is a supervised machine learning algorithm, i.e., it relies on labeled input data to learn a function and produces output according to the new unlabelled data. That is an instance-based learning method. It has been considered one of the simplest of all machine learning algorithms. The nearest Neighbour classification relies on two parameters, i.e., nearest distance metric (Euclidean Distance) and neighbors' number (k variable). Algorithm 1 shows the KNN classification pseudocode [38].

In a supervised machine learning algorithm, SVM is one of the most straightforward ways to classify data. Simply put, the main idea of classifying the data using SVM is to separate the two data sets with a hyperplane and maximize the margin between the two data classes. Algorithm 2 shows the SVM classification pseudocode [38].

Algorithm 1. KNN Algorithm

Algorithm 2. SVM algorithm

Algorithm 3. Random forest algorithm

Algorithm 4. Decision tree algorithm

Random forest is a supervised learning algorithm. It can be used for classification and is the most flexible and easy-to-use algorithm. Random forests build decision trees on randomly chosen data samples, obtaining predictions from each tree and selecting the optimal solution through voting. The process involves selecting random samples from a provided dataset and constructing a decision tree for each sampled subset.

Then, use each decision tree to generate a prediction result. Finally, choose the prediction result that receives the highest number of votes as the final prediction.

RF is composed of many binary decision trees and is used for diverse purposes such as regression, classification, and other applications via generating a large number of decision trees during the time of training [8]. Algorithm 3 [38] demonstrates the Random Forest classification pseudocode.

A Decision Tree is a supervised learning technique used for data categorization, and it has two nodes. Decision nodes are employed for decision-making, featuring multiple branches, while leaf nodes represent the outcomes of those decisions and lack additional branches. Algorithm 4 shows the Decision Tree classification pseudocode [38].

This research has analyzed different machine learning algorithms and their advantages and limitations. For instance, the K-Nearest Neighbors (KNN) algorithm works well in multi-class classification scenarios but needs help with datasets with many features due to dimensional complexity. On the other hand, Support Vector Machines (SVM) are proficient in handling linear and non-linear feature-to-target relationships using various kernel functions. However, selecting the proper kernel and fine-tuning the model parameters is complex. Decision trees are good at identifying non-linear patterns, but they can become unstable due to minor changes in the input data, resulting in significantly different tree structures. Conversely, Random Forests prevent overfitting issues by utilizing the power of multiple decision trees, but they can be biased towards dominant classes in unbalanced datasets. According to the results and discussion, the Random Forest algorithm has performed better in classification accuracy than other algorithms analyzed in this research.