R. Rathna*
| V. Maria Anu | Siddhartha Mishra
© 2024 The authors. This article is published by IIETA and is licensed under the CC BY 4.0 license (http://creativecommons.org/licenses/by/4.0/).
OPEN ACCESS
According to the World Health Organization, 430 million individuals, or more than 5% of the world's population, need hearing loss rehabilitation. They also estimate that by 2050, one in ten individuals, or over 700 million people, would suffer hearing loss, that is incapacitating. Despite these difficulties, technological developments have made it feasible to enhance these people's quality of life. The Real-time Smart Auditory Assistive Wearable (RESAAW), is a revolutionary gadget that aids people with hearing problems by employing omnidirectional microphones to record real-time surrounding noises. In the case of abrupt or harmful noises, the gadget gives the user tactile input while employing advanced digital signal processing and machine learning models to distinguish vital sounds from background noise. The goal of RESAAW is to use 4 omnidirectional microphones that constantly pick up noises in the vicinity of the individual wearing it and filter the noises using digital signal processing techniques ML models. There are people with different degrees of hearing loss. This system would be a boon to them in their work places.
microprocessor, digital signal processing, machine learning, vibration and sensor
Assistive devices for the hard of hearing have come a long way in recent years, and there is ongoing research in this field to further improve their effectiveness and accessibility. These devices aim to improve communication, independence, and quality of life for people with hearing loss. The area of your hearing that is affected determines the sort of hearing loss you experience. Hearing loss can be classified into three categories:
When sound cannot be properly transmitted from the outside or middle ear to the inner ear, conductive hearing loss develops. Because of this, the level of sound that enters the inner ear may be reduced, making it challenging to hear speech or subtle noises. Conductive hearing loss can affect one or both ears, and it can be either temporary or permanent. There can be several reasons [1] for conductive hear loss.
Depending on how severe the problem is, conductive hearing loss can present with a variety of symptoms. Some typical signs include hearing quiet noises or speech with difficulty, distorted or muddled hearing, the experience of having a full or blocked ear, discomfort or pain in the afflicted ear and hearing ringing or tinnitus.
In case of sensorineural hearing loss, when the auditory nerve that transmits signals to the brain or the inner ear (cochlea) is damaged, hearing loss known as sensorineural hearing loss results.
Depending on how severe the problem is, sensorineural hearing loss can present with a variety of symptoms. Some typical signs include having trouble hearing speech or quiet noises, especially in busy situations, the requirement to increase the radio or television's loudness, ringing in the ears, called tinnitus, speech is difficult to comprehend, especially when spoken by toddlers pose a challenge in differentiating sounds, especially ones with identical frequencies.
The primary objective of this system is to help save the lives of the people with hearing loss problem. There are fatalities happening every year because of their hearing loss. Recently a middle aged woman who is having a hearing loss died on the spot while crossing the track inside a railway station. The train was not even faster but the impact of hitting her caused the death because there was no big noise. Such noise less movements can be identified and alert will be sent through the system proposed here.
Even though many small and ergonomic assistive devices are available for people with this problem, they are all expensive. They are not affordable for 50% of people (falling under below poverty line) in a developing country like India. Resulting in many children growing up without any assistive devices and getting accustomed to the hearing loss problem in economically poor background areas. This leads to denial of education to them in a normal school. The work proposed here is about an assistive device which is economically affordable and made up of very simple technology.
The need for having a device to assist in hearing for the people with hearing loss problem was there from 15th century. The first one was developed in the late 18th century. It was a device which is trumpet shaped. Some of them are simple cones. It would amplify the noise fed into it and gives the output, if kept near the ear. It was actually developed for the kings and for people in higher position. Later in the 19th century, electrical based, acoustic based devices were developed. Then many innovative hidden hearing assistive devices were also developed. Electronic and computer based hearing aids were developed only after 1960s. It took such a long journey to reach the state of device what is available now. The one available now is small, aesthetic, weight-less but expensive.
There has been a lot of study and development put into hearing-impaired people's assistive technology. The progress of technology has led to the production of a broad variety of tools that can enhance social interaction and communication for people with hearing loss. A hearing aid is one of the most well-liked pieces of assistive technology for those who are hard of hearing.
Cochlear implants are another type of assistive technology that can be very helpful for those with severe or profound hearing loss. Cochlear implants [2], which are electrical devices surgically placed in the inner ear, can directly stimulate the auditory nerve. People whose hearing loss is too severe to be helped by hearing aids frequently use them. A further category of assistive technology for those with hearing impairments is personal amplifiers, which are small, portable devices that may be worn in the ear or carried in the pocket that can amplify sound in noisy environments. Furthermore, FM systems, which transmit sound directly to hearing aids or cochlear implants, can help people understand speech better in noisy environments.
In addition to these assistive devices, there has been much research into other technologies that can help people with hearing loss communicate more effectively. For instance, speech recognition technology has advanced significantly in recent years, and there are now apps and software programs that can instantly convert spoken words into text, enabling persons with hearing loss [3] to participate fully in conversations. People with severe hearing loss may benefit from cochlear implants, which can restore their sense of hearing. As well as creating novel methods for implanting and programming these devices, researchers are striving to enhance the design and performance of these devices. For example, scientists are attempting to create new electrodes that can be implanted deeper into the cochlea, enabling more accurate stimulation of the auditory nerve and maybe higher sound quality.
The creation of new, more sophisticated, and efficient varieties of hearing aids is the subject of another field of study. This involves the creation of hearing aids that may adjust to a user's particular hearing requirements using machine learning algorithms. In order to optimize a hearing aid's amplification and noise reduction settings, machine learning algorithms may be employed to determine a person's unique hearing loss, listening environment, and listening preferences. Additionally, researchers are aiming to create hearing aids that can link directly to other devices, such as smartphones, enabling remote control, programming, and audio streaming to the hearing aid.
Hearing loss can be caused by several conditions, including predisposition to genetic diseases, trauma, loud noise exposure, and ageing. Some people are born with hearing disabilities, while others develop it later in life. Tinnitus can also be caused by medical diseases such as otitis media or Meniere's disease. Furthermore, chronic exposure to high-intensity noise [4], such as that found in industrial facilities or via the use of traditional high-frequency personal hearing aids, can damage the fragile tissues of the inner ear, eventually leading to hearing disabilities. With RESAAW, the aim is to provide a simple prototype device to assist the hearing challenged individuals.
Some of the previous work described [5] and proposed an IoT based communication assistant application for the hearing challenged individuals, by utilizing a microcontroller, pulse sensor, and interfacing electret microphone amplifier. Their web-based application can provide information about the strength of the sound produced and the text of the conversation of the interlocutor. In their research they have developed a sensor device consisting of a NodeMCU V3, a pulse sensor, and connecting electret microphone amplifiers, to create an Internet of Things (IoT)-based communication assistant application for the hearing impaired. To track the volume of sound people with hearing loss talk with while they speak, sensors regularly relay data to a computer.
According to Angdresey et.al. [6], the study examined the ways in which prolonged exposure to sound affects the brain. The study focused on hearing challenged individuals who had longterm usage of hearing devices despite being born with hearing disabilities. They matched their brain activity to that of individuals with normal hearing. While having the activity of their brains monitored, both groups experienced sensations on their hands. The area of the auditory cortex, which processes sound, was more active in the hearing challenged group. The reason, according to the researchers, is that the hearing challenged individuals' brains reacted to the vibrations differently from those of the hearing group since they were accustomed to experiencing vibrations from their hearing aids.
The researchers [7] proposed a wireless calling bell system for the hearing challenged individuals and mute called WiBeD2. It is a prototype that is worn on the wrist and is based on the idea of a wearable gadget. The system includes message acknowledgement switches, an LED indication, and a vibrator. The wireless device vibrates when the calling bell is pressed, and an LED indicator appears. The system makes use of wireless communication technologies and wearable gadgets to enable accessibility for the hearing challenged individuals and the mute. It incorporates switches for message acknowledgment, LED and LCD displays, and vibration.
Also, the purpose of the study was to evaluate how 15 preschool-aged severely retarded hearing and visually impaired children responded to aural stimulation [8]. The difference in the children's operant choices between no sound and music performed at 80 dB SPL was assessed by the researchers. Those kids who had trouble answering were given vibratory help. It was discovered that the assistance was essential in enabling some of the kids to consistently respond to the auditory stimuli. The study's overall findings highlighted the significance of adopting a variety of tools and methods to support the listening and communication skills development of kids with hearing and vision impairments.
In the research [9], the paper describes a gadget that can help those who are visually, vocally and hearing challenged. It is based on Google API and Raspberry Pi technology. The gadget has the capacity to take pictures and use image-to-text conversion technologies to turn them into text. The captured text is then transformed into voice output so that visually impaired people can hear it. The gadget records spoken words with a microphone and translates them into text format, which is then shown on the screen for individuals who have trouble hearing. The device can help in translating documents, books, and other materials, which can enhance the quality of life for individuals with disabilities.
According to the study [10], the human nervous system may adjust to sensory deprivation by creating new methods of information processing. The auditory cortex, for instance, may begin to process tactile (touch) information in persons who are hard of hearing from birth. They discovered that the hearing challenged individuals had improved tactile sensitivity because they were better able to recognize changes in the vibratory stimuli. This increase in sensitivity might be the result of more attention being paid to the stimuli or neuronal plasticity, where the brain creates new pathways to receive tactile information.
In the study [11], the proposed wearable gadget will alert hearing-impaired individuals when someone is attempting to grab their attention. The gadget vibrates to inform the user after identifying the user's name or common words like "hey" or "hello" using sound detection technology. The Voice Recognition Module (v3) sensor, which is used by the system, can identify the user's name when they are called by a third party. A vibration motor is then used to turn the sound signals into vibrations.
In the study [12], the created wearable device contains a number of LED outlets, vibration motors, and microphones. The sound data collected by the microphones is analyzed to determine the kind of sound. The device then employs particular vibration frequencies to let the user know where the sound is coming from. The most effective feature extraction and classification techniques were chosen using this procedure. The prototype gadget was then put to the test using these chosen techniques, with the KNN [13] classification and SVM [14] attaining 93% and 94% success rates, respectively. With SVM and two of the best feature approaches, the system had a 98% success rate up close. Overall, this system aims to help hearing-impaired individuals detect sounds and their direction through vibrations and LED indicators.
In this proposed research work, the focus is given more on using vibration as a signal for the hearing challenged individuals. According to medical research, vibration technology has proven to be a useful tool for individuals who are hard of hearing. Vibration is a powerful tool for individuals who are hard of hearing, as it allows them to perceive sound in a different way than those who can hear normally. Research has shown that hearing challenged individuals are able to perceive vibrations in a more sensitive and nuanced way than those who can hear. One of the main reasons that vibration is better perceived by hearing challenged individuals is that it bypasses the auditory system entirely.
When individuals hear sound, it is first processed by the ear and then transmitted to the brain for interpretation. However, for individuals who are hearing challenged, this process is not possible. Instead, vibrations are directly transmitted to the brain, bypassing the ear and allowing for a more direct and accurate perception of sound. Another reason that vibration is better perceived by hearing challenged individuals is that it can provide a more complete and holistic representation of sound. When individuals hear sound, they typically only hear a small portion of the sound waves. However, when vibrations are used, individuals can feel the entire sound wave, providing a more complete representation of the sound. In addition to these benefits, vibration technology can also improve the overall quality of life for hearing challenged individuals. For example, the use of vibrating alarms can help challenged individuals wake up on time, and vibrating doorbells can help them know when someone is at the door. This can greatly improve their independence and overall ability to navigate their daily lives. Using this as the basis of this proposed work, the ability of hearing challenged individuals to differentiate between vibrations of different intensity has been explored. Hearing challenged individuals have a unique ability to differentiate between vibrations of different intensity, which is better than normal people.
Research has shown that the brains of hearing challenged individuals have a heightened sensitivity to vibrations [15], which allows them to perceive a wider range of vibrations and differentiate between vibrations of different intensity more accurately than those who can hear normally. The brain can rewire itself to adapt to new experiences and environments. For hearing challenged individuals, the brain rewires itself to process vibrations differently, allowing for a more accurate perception of sound [16]. This neuroplasticity allows the brain to process vibrations in a more refined and sensitive manner. Another reason that hearing challenged individuals are better at differentiating between vibrations of different intensity is because they rely more heavily on their sense of touch [17].
Research has shown that the neural pathways in the brains of hearing challenged individuals are more developed and specialized for processing vibrations than those of individuals who can hear normally.
When a noise is considered not part of the general white noise of the environment like a sudden noise or a progressively incremental noise, the RESAAW provides the user with vibrational feedback from the specific direction where this noise was picked from.
3.1 Architecture
The proposed system for RESAAW is a very simple one. It works using very simple techniques of digital signal processing and Machine Learning. The device itself is a very simple embedded system device using an Arduino Uno as its processor. In our proposed system, we are using omnidirectional microphones that would pick up noises in the surroundings and then use digital signal processing techniques to filter the sounds. When a sudden noise or a constant approaching noise is heard, the machine learning technique used in RESAAW will alert the user through vibrations of vibrational motors. The received noise is compared with historic data of noises and XGBoost [18] algorithm is used for prediction. It is an ensemble algorithm which does not depend upon single model’s prediction. It ensembles the result of combined models. This algorithm is computationally effective for data received from such low level devices. The basic idea of the proposed system can be pictured through this flow chart in Figure 1.
Figure 1. Architecture of RESAAW
3.1.1 Components of REESAW
Arduino Uno Microcontroller: In the Arduino family, the ATmega328P microcontroller chip is the basic building block of the open-source Arduino Uno microcontroller board. Due to its relatively low cost, straightforwardness, and versatility to construct numerous electrical designs, the Arduino Uno (Figure 2) is a widely recognized and frequently used development board.
Figure 2. Arduino Uno
Electret Omnidirectional Microphones: A type of microphone that can equally capture sound from all directions is the omnidirectional electret Figure 3. They are often utilized in a wide range of settings, including recording studios, meeting spaces, and the design of numerous technological equipment.
A condenser microphone that employs a permanently charged electret material as the diaphragm is known as an electret microphone. The omnidirectional power microphone has a spherical diaphragm, which is sensitive to the sound waves that work in all directions. The omnidirectional power microphone has a spherical diaphragm, which is sensitive to the sound waves that work in all directions. The other plate of the diaphragm is only a short distance from the other plate of the back plate. When the sound wave hits, the film will vibrate. When the diaphragm is vibrated, the separation of the two plates of the capacitor is different, and the tiny electrical signal will generate corresponding to the sound waves.
Figure 3. Electret microphone
A condenser microphone that employs a permanently charged electret material as the diaphragm is known as an electret microphone. The omnidirectional power microphone has a spherical diaphragm, which is sensitive to the sound waves that work in all directions. The omnidirectional power microphone has a spherical diaphragm, which is sensitive to the sound waves that work in all directions. The other plate of the diaphragm is only a short distance from the other plate of the back plate. When the sound wave hits, the film will vibrate. When the diaphragm is vibrated, the separation of the two plates of the capacitor is different, and the tiny electrical signal will generate corresponding to the sound waves.
Figure 4. Vibration motor
Vibration Motor: A motor that vibrates while it turns is known as a DC motor vibrator. It comprises of an eccentric weight, sometimes referred to as a vibrating mass, and a DC motor Figure 4. The eccentric weight spins when the motor is turned on, which causes it to move in a circular manner. There are audible or sonic vibrations produced by the weight as it travels. The DC motor vibrator operates based on rotational motion and the mass's inertia. A force that oscillates back and forth is produced by the mass as it travels in a circular motion. The vibrations that are sensed or heard are created by this oscillation. The motor's speed and direction control the vibrations' frequency and intensity. The frequency and power of the vibrations will grow as the motor speed increases.
3.1.2 Credo used in RESAAW
Here is where the digital signal processing techniques
By using this algorithm, REESAW will update and store the current data about object movements and their predicted values.
Figure 5. RESAAW
As mentioned earlier, RESAAW is an assistive device Figure 5. to help the hearing-impaired people. It’s working is based on the data received by 4 omnidirectional electret microphones and the output is produced through 4 DC Vibrator Motor actuators.
RESAAW provided promising results. Showing a clear pattern of how the 2 SMA [19] filters are working together and how the condition is being triggered to vibrate the microphones. The output is being visualized in the following screenshots of the Serial Plotter in Figure 6 and Figure 7.
Figure 6. Vibrating when a loud sudden noise is heard
Figure 7. Closer look at the 4 lines
In Figure 6 and Figure 7, we can see that the blue line is a reference base line always fixed at ‘0’. The Yellow line is the line signifying the vibration of motor.
When the orange line which signifies a sudden loud noise is heard, we can see a vibration is heard. The green line signifies the varying baseline as per the sound levels in the environment of RESAAW.
In Figure 8 and Figure 9, the blue line represents the reference base line which is always fixed at ‘0’. The Yellow line is the line signifying the vibration of motor. When the orange line which signifies an approaching noise is heard, a continuous vibration can be observed.
The green line signifies the varying baseline as per the sound levels in the environment of RESAAW which is also increasing as the average sound levels of the surrounding are increasing.
From the figures it is clear that the RESAAW prototype is working as expected. Whenever a sudden noise is heard, the processed SensorValue shows a steep spike which implies the first SMA filter is working fine and there are steep changes in the value when a sudden noise is heard. Whenever a loud and approaching noise is heard, the processed SensorValue stays continuously above the baseline signifying the SMA filter is working for longer duration of high sound exposures. The vibrator is also working as expected.
Figure 8. Vibrating when an approaching noise is heard
Figure 9. Close look at constant vibration for approaching noise
Whenever a steep sound is heard, a vibration for 1 second is produced but when a continuously high sound is heard, the vibrator keeps rumbling for a greater time period. Not only it worked for simple real time samples but also for the data used from CHiMe data set [20] containing variety of domestic sounds. The false positives were 3% and the true negatives were 23.33%.
Many research works are already going on in this assistive technology. Also many kinds of wearable assistive devices are in the market (Like smart cane and smart glasses) now. They use ML and are able to attain an accuracy between 90 to 95%. But most of them uses images for identifying the approaching objects and they are all computer vision based. In the work presented here, only from the digitized sounds, ML is used and alert is given to the user. It produces an accuracy of 82%. It can be customized as a smart cap in the future.
The RESAAW initiative aimed at aiding individuals with hearing impairments has proven to be a triumph. The RESAAW system has been developed with the aim of enhancing the awareness of individuals with hearing impairments. Empirical evidence from our testing indicates that the system is functioning as intended. RESAAW's innovative design facilitates ease of use and has the potential to enhance the quality of life for individuals experiencing hearing impairment. The potential advantages of the Resilience, Empowerment, and Advocacy in Action Workshop (RESAAW) are noteworthy, and it is our aspiration that it will be embraced by those who require it the most. In future, RESAAW can be designed as a wearable cap that is powered by batteries to make it easier to wear and carry. And it can be made IoT enabled. The vibration patterns can also be set for even more classification of noises. The design is made in such a way, that it can be easily integrated with home automation system using IoT with very little modification.
[1] Haile, L.M., Hosseini-Araghi, M.H., Johnson, N.L., Thomson, M.E., VanderWerf, F., McPhedran, R., Vos, T. (2021). Hearing loss prevalence and years lived with disability, 1990-2019: Findings from the global burden of disease study 2019. The Lancet, 397(10278): 996-1009. https://doi.org/10.1016/S0140-6736(21)00516-X.
[2] Hamlin, L. (2022). Hearing assistive technology. The Hearing Journal, 75(9): 12. https://journals.lww.com/thehearingjournal/toc/2022/09000.
[3] Fontan, L., Álvarez-Móran, A., Fernández-Carballo, J.C. (2022). Using automatic speech recognition to optimize hearing-aid time constants. Frontiers in Neuroscience, 16. https://doi.org/10.3389/fnins.2022.779062
[4] Le, T.N., Straatman, L.V., Lea, J., Westerberg, B. (2017). Current insights in noise-induced hearing loss: A literature review of the underlying mechanism, pathophysiology, asymmetry, and management options. Journal of Otolaryngology-Head & Neck surgery, 46(1): 41. https://doi.org/10.1186/s40463-017-0219-x
[5] Dash, G.S., Rout, S., Singh, O. (2016). WiBeD2: A communication aid for deaf and dumb. In 2016 International Conference on Information Communication and Embedded Systems (ICICES), Chennai, India, pp. 1-4. https://doi.org/10.1109/ICICES.2016.7518846
[6] Angdresey, A., Masala, I.V., Kumenap, V.D., Sumampouw, M.G., Dame, K.A., Riady, I.D. (2021). A communication assistant application for the deaf. In 2021 Sixth International Conference on Informatics and Computing (ICIC), Jakarta, Indonesia, pp. 1-6. https://doi.org/10.1109/ICIC54025.2021.9633011
[7] Auer, E.T., Bernstein, L.E., Sungkarat, W., Singh, M. (2007). Vibrotactile activation of the auditory cortices in deaf versus hearing adults. NeuroReport, 18(7): 645-648. https://doi.org/10.1097%2FWNR.0b013e3280d943b9
[8] Friedlander, B.Z., Silva, D.A., Knight, M.S. (1973). Selective responses to auditory and auditory-vibratory stimuli by severely retarded deaf-blind children. Journal of Auditory Research, 13: 105-111. https://psycnet.apa.org/record/1975-31904-001.
[9] Karmel, A., Sharma, A., Pandya, M., Garg, D. (2019). IoT based assistive device for deaf, dumb and blind people. In 2019 International Conference on Information Communication and Embedded Systems (ICICES), 16: 259-269. https://doi.org/10.1016/j.procs.2020.01.080
[10] Levänen, S., Hamdorf, D. (2001). Feeling vibrations: Enhanced tactile sensitivity in congenitally deaf humans. Neuroscience Letters, 301(1): 75-77. https://doi.org/10.1016/S0304-3940(01)01597-X
[11] V, D.K., R, Y., Rangani, T.L., Sen, A. (2022). IoT based divyang assistant technology: Your hearing support. In 2022 International Conference on Electronics and Renewable Systems (ICEARS), Tuticorin, India, pp. 474-481. https://doi.org/10.1109/ICEARS53579.2022.9752307
[12] Yağanoğlu, M., Köse, C. (2017). Wearable vibration based computer interaction and communication system for deaf. Applied Sciences, 7(12): 1296. https://doi.org/10.3390/app7121296
[13] Sajja, T.K., Kalluri, H.K. (2020). A deep learning method for prediction of cardiovascular disease using convolutional neural network. Revue d'Intelligence Artificielle, 34(5): 601-606. https://doi.org/10.18280/ria.340510
[14] Cervantes, J., Garcia-Lamont, F., Rodríguez-Mazahua, L., Lopez, A. (2020). A comprehensive survey on support vector machine classification: Applications, challenges and trends. Neurocomputing, 408: 189-215. https://doi.org/10.1016/j.neucom.2019.10.118
[15] Flores Ramones, A., del-Rio-Guerra, M.S. (2023). Recent developments in haptic devices designed for hearing-impaired people: A literature review. Sensors, 23(6): 2968. https://doi.org/10.3390/s23062968
[16] Hribar, M., Šuput, D., Battelino, S., Vovk, A., Grosman, M. (2021). Structural brain alterations in prelingually deaf. NeuroImage, 220: 117042. https://doi.org/10.1016/j.neuroimage.2020.117042
[17] Pieniak, M., Michalak, M., Mazur, K., Derezińska, E., Jabłoński, M. (2020). Self-rated sensory performance in profoundly deaf individuals. Do deaf people share the conviction about sensory compensation? Journal of Sensory Studies, 35(4). https://doi.org/10.1111/joss.12572
[18] Tarwidi, D., Pudjaprasetya, S.R., Adytia, D., Apri, M. (2023). An optimized XGBoost-based machine learning method for predicting wave run-up on a sloping beach. MethodsX, 10: 102119. https://doi.org/10.1016/j.mex.2023.102119
[19] Bose, B., Dutta, J., Ghosh, S., Pramanick, P., Roy, S. (2018). Smartphone based system for real-time aggressive driving detection and marking rash driving-prone areas. In 19th International Conference on Distributed Computing and Networking (Workshops ICDCN '18), New York, NY, USA, pp. 1-6. https://doi.org/10.1145/3170521.3170549
[20] Foster, P., Sigtia, S., Krstulovic, S., Barker, J., Plumbley, M.D. (2015). CHiME-Home: A dataset for sound source recognition in a domestic environment. In 2015 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA), New Paltz, NY, USA. https://doi.org/10.1109/WASPAA.2015.7336899
