Addressing Hearing Impairments through Machine Learning: A Review of Sound Detection and Assistive Technologies ()
1. Introduction
In daily life, humans face several dangers that often require constant vigilance and caution to deal with surrounding risks in general. This necessitates attention, identification of potential hazards, determination of the type of risk, and subsequent action accordingly. Risks vary in terms of response speed and the level of danger they pose. Some, like responding to fire alarms in a building, demand swift movement and evacuation, potentially even saving lives.
One of the most significant obstacles that may impede a swift response is the loss of hearing. Deaf or hard-of-hearing individuals face additional risks, such as an inability to respond to potential warnings and alerts. They heavily rely on others in times of danger, leading to a loss in response speed and the ability to promptly handle similar risks. Furthermore, beyond difficulty in hearing and identifying risks, they struggle with oral communication in their daily lives, especially in times of potential danger.
Recent data from the World Health Organization (WHO) highlights a growing global trend in hearing impairment. Currently, over 430 million people, more than 5% of the global population, suffer from hearing loss, whether congenital or acquired. By 2050, this number is expected to exceed 700 million, meaning one in ten people will experience disabling hearing loss [1]. This group is identified as the second-largest among individuals with disabilities in the General Census of Population and Housing [2].
In response to these growing needs, rapid advancements in computer science—particularly in artificial intelligence (AI)—have introduced powerful tools to support the hearing-impaired community. Technologies such as sound recognition and speech-to-text systems have significantly improved their ability to detect surrounding hazards and communicate in daily life. While early assistive solutions focused on limited alerts (like vibrations or flashing lights), recent AI-based systems use machine learning to analyze and classify environmental sounds in real time. These modern systems represent a leap forward, offering both greater functionality and adaptability.
So, in this paper, it will serve as the literature review, elucidating the key concepts central to understanding how machine learning can contribute and utilize to enhance comfort and safety environment for deaf and hard of hearing people. It commences with a concise overview of various assistive technologies. Also, provides an exploration of sound detection technologies and comprehensively covers sound detection within machine learning, encompassing preprocessing, feature extraction, model training and classification, real-time aspects, and associated challenges and constraints. In addition, a grasp of speech recognition and automatic speech recognition is established. To conclude, this comprehensive review presents an examination of research papers and applications offering insightful reviews.
2. Assistive Technologies to Hearing Disabilities
Assistive technologies (AT) have been among the most significant advancements in the past 20 years for helping individuals with disabilities, including those with hearing impairments. According to ISO 9999:2016 and UNE-ISO 9999:2017, AT encompasses any product—whether a device, piece of equipment, instrument, or software—specifically designed to improve the engagement and functionality of individuals with disabilities. These technologies assist, support, train, measure, or substitute for bodily functions, aiming to prevent impairments, activity limitations, or participation restrictions [3].
For those with hearing impairments, AT includes hearing aids, communication systems, low-tech devices, cochlear implants, and specialized software and hardware that enhance hearing and communication abilities. These tools not only promote independence and well-being but also prevent secondary health issues and provide socioeconomic benefits by reducing healthcare costs and stimulating economic growth [3].
In communication for the deaf and hard of hearing, significant technological progress has been made. Text telephones (TTY) and telecommunications devices have been crucial in enabling telephone communication for this community, with communications assistants helping to bridge the gap between hearing-impaired and standard telephone users by relaying messages . Recent innovations include sign recognition systems for interpreting sign language gestures and Personalized Emergency Response Systems that enhance safety through sensor-based alarms [4] [5]. Mobile applications have also become a breakthrough, empowering the hearing-impaired to live more autonomously and communicate effectively within their communities . These technologies cater to the diverse needs of individuals with hearing impairments across various areas of life.
3. Sound Detection Technologies
Sound detection technology has seen growing interest due to recent advancements that have made it more reliable and precise. These developments enable machines to mimic human hearing and understand context, allowing for applications such as smartphone alerts [6], diagnosing conditions like coughing [7], improving security by identifying threats, aiding in cataloging audio archives [8], and enhancing safety monitoring on construction sites [9]. These technologies provide critical situational awareness, especially in environments beyond visual or attentive reach.
Sound detection involves identifying sound events in a continuous audio signal and analyzing them using various methodologies to extract relevant information, depending on the intended application. This process typically involves multiple techniques related to audio signal processing or machine learning. For example, traditional computational analysis systems extract specific acoustic features from an input signal, which can then be categorized and detected using supervised classifiers like neural networks. Developing sound detection applications requires defining several factors, such as the nature of the application, technological constraints, desired complexity and precision, and data availability [10].
Sound event detection systems are usually customized for specific tasks and environments, requiring a combination of different techniques and processes. The implementation of sound detection technologies involves integrating various methods tailored to specific purposes. The following section will explore the core processes and key techniques used in sound detection technologies, particularly those involving machine learning, in different environments.
4. Machine Learning-Based Sound Detection
Machine learning has significantly advanced numerous domains, as highlighted by Rebala et al. They have succinctly defined the core processes of sound detection within the broader machine learning context. They characterize machine learning as a computer science discipline dedicated to automating solutions for intricate problems that defy conventional programming methods. Traditional programming necessitates meticulous design and code implementation, posing challenges for tasks such as character recognition or sound event detection. In contrast, machine learning algorithms acquire knowledge from labeled data, bypassing the need for explicit rules. These algorithms excel at solving complex problems with greater accuracy and objectivity compared to rules crafted by humans. Their approach involves creating a model from a dataset and subsequently predicting labels for new data points [11].
Therefore, in order to improve the detection of sound events across various applications, it is contemporary to employ machine learning methods and techniques, whether in real-time or not. It is also essential to grasp the key challenges and constraints associated with machine learning-based sound detection. So, the following sections will discuss more about these topics.
4.1. Processes and Technics for Sound Detection
To gain a deeper understanding of the fundamental processes and methodologies utilized in machine learning for sound detection, the two referenced papers [12] [13] provide insights into the core stages of sound detection technologies or sound event detection systems. These stages comprise three key components: 1) Preprocessing, 2) Feature Extraction, and 3) Model Training and Classification. Various papers will demonstrate a range of machine-learning techniques for sound detection, with supervised learning emerging as the dominant approach in tackling the sound event detection task.
4.1.1. Preprocessing Step
The initial stage in sound event detection entails the application of various techniques to enhance the quality of the audio data before feature extraction. This step is necessary because raw audio data cannot be directly employed as input for machine learning-based classification. The rationale behind this necessity lies in the presence of signal redundancy, which must be addressed. Preprocessing activities typically encompass tasks such as noise reduction, equalization, low-pass filtering, and segmenting the original audio signal into audio and silent events to facilitate subsequent feature extraction [12] [13].
Data Collection: Effective sound detection systems hinge on understanding sound data and context, especially in supervised learning, where the training data must closely resemble the intended application scenario. As sound event detection can cover a broad range of sound classes and environments, no single dataset or acoustic model fits all scenarios. Instead, multiple datasets are curated to address specific challenges, with dataset size often reflecting the complexity of the labels. Access to diverse datasets is crucial for training models that can adapt to different environments and sound events [13].
Three research papers explore gunshot-related sound detection systems using various datasets, achieving different accuracy levels. The first study uses a sound event recognition model trained with 692 samples from sources like Freesound.org and YouTube, achieving an accuracy of 77.32% in indoor event classification [14]. The second paper introduces a hybrid algorithm for detecting gunshots in indoor settings, with data collected from online sources and real-world locations like shopping malls and universities, achieving accuracy between 91.65% and 94.97%, depending on the classifier [15]. The third paper presents a novel approach for recognizing environmental sounds across various settings, using a dataset of 1000 sounds across 10 categories, including gunshots, and achieving up to 92.22% classification accuracy [16].
Audio Signal Preprocessing is pivotal for the effectiveness of machine learning algorithms, particularly in creating generalized predictive models for classification tasks [17]. This phase involves a variety of techniques to prepare the audio signal for feature extraction, with the choice of methods depending on the sensitivity and accuracy requirements of the specific application or system.
Normalization is an essential technique in data analysis, especially when handling data from different sources with varying measurement scales. Dalwinder and Birmohan define data normalization as a process that resizes or transforms raw data so that each feature contributes uniformly. For example, when data includes both age (in years) and height (in centimeters), normalization ensures consistency in the magnitude of these parameters [17] [18].
In signal processing, Noise Reduction and Filtering are two fundamental methods. Noise refers to unwanted disturbances within a specific frequency range, such as electric waves and random variations [19]. Noise reduction is crucial for cleaning data that is susceptible to noise interference [18]. Filtering is closely related and focuses on eliminating noise, often using low-pass or high-pass filters to remove high-frequency noise or highlight specific data features [20]. These methods are particularly useful in medical fields like Electrocardiogram (ECG) signal analysis. ECG signals, which record heart rates, are vital for investigating abnormal heart functions, such as arrhythmias and conduction disturbances. Filters like low-pass, high-pass, and Butterworth filters are employed to preprocess these signals by effectively removing high-frequency noise, with Butterworth filters being particularly effective [21].
Another important preprocessing step is Silence Removal, which addresses the presence of complete silence at the beginning, end, or within audio signals. This process applies a specific threshold to remove unvoiced portions that lack relevant data, retaining only the voiced sections [22]. In studies related to cough sound detection, silence removal is crucial as it allows the focus to remain on sound events, thereby improving detection accuracy [23]-[25].
In practice, sounds often overlap or occur sequentially, requiring effective segmentation for classification. Segmentation prepares audio signals for feature extraction by dividing them into distinct segments based on temporal proximity and setting thresholds to determine the relevance of sound segments [26].
The final preprocessing method is Feature Windowing, which treats non-stationary signals as quasi-stationary by sliding a window over the entire signal for comprehensive analysis . Unlike traditional acoustic analysis systems that divide sound recordings into fixed-sized windows, contemporary methods adapt the window size to the signal’s characteristics [27] [28].
Segmentation and feature windowing are closely related but serve different purposes. While segmentation focuses on creating meaningful signal segments, feature windowing concentrates on extracting features within these windows. For instance, Baughman et al. used a peak detection method to identify specific sounds in a tennis match recording, isolating the sound of interest within a single window. This technique is valuable for classifying acoustic events using machine learning. However, if the sound of interest spans two different windows, classification accuracy may be reduced [28].
Ultimately, these preprocessing methods lead to feature extraction, where the extracted features are stored in a database for training the classifier, a topic to be discussed in the next step.
4.1.2. Feature Extraction Step
Feature extraction is vital in audio content analysis as it involves creating a numerical representation, or feature vector, that captures key acoustic characteristics of audio segments [12] [13]. This vector is foundational for various audio analysis and information extraction algorithms [29], as it condenses extensive data into a more manageable format while retaining essential information [30].
Time-frequency analysis techniques are commonly used in feature extraction to focus on the signal’s frequency domain, breaking the signal into overlapping frames to track frequency distribution changes over time [30]. These techniques, such as Mel frequency cepstral coefficients (MFCCs), Log-Mel energies, and Spectrograms, divide the signal into smaller segments and calculate the frequency content for each. The resulting magnitude spectrum shows energy distribution over frequency for each segment, allowing for the computation of a concise set of features that capture fundamental spectral characteristics. This compact feature set is preferred for machine learning algorithms, as it maintains informativeness while reducing complexity, making it widely applicable in signal processing tasks [29].
Different sound extraction features are used depending on the system and its performance needs. Common methods include log-mel energies, MFCCs, spectrograms, and constant-Q filterbank-based features. For instance, Jain et al. developed ProtoSound, an interactive system that enhances sound awareness for deaf or hard-of-hearing individuals. ProtoSound personalizes sound recognition models using user recordings and log-mel spectrogram features, significantly improving performance when integrated with deep convolutional neural networks (CNN) [31].
Deep Neural Networks (DNNs) have been used to classify cough sounds by extracting MFCC features. Liu et al. [32] reported a DNN with MFCC features achieving 90.1% accuracy for positive cases and 85% for negative cases, while Amoh and Odame [33] attained 86.8% accuracy for positive cases and 92.7% for negative cases using a similar approach.
In speech-based emotion recognition, extensive reviews have compared different approaches. Studies using the RAVDESS, Emo-DB, and IEMOCAP datasets found that Log-Mel spectrogram features outperform MFCCs, challenging their prevalent use in this field [34] [35]. Other feature types are also effective for sound event recognition. For example, Sing et al. analyzed constant-Q filter bank-based time-frequency representations, offering superior frequency resolution at low frequencies compared to MFSC. Wang et al. used a model based on discrete Fourier parameters, where frequency harmonics and their derivatives effectively distinguished emotion classes. Additionally, Badshah et al. introduced a method combining spectrograms and CNN for sound event recognition, achieving promising results in emotion prediction [36]-[38].
4.1.3. Model Training and Classification Step
During this phase, the system learns to correlate extracted audio signal features with specific class labels to develop a model for categorizing audio recordings into predefined classes. For instance, a sound scene classification system might categorize recordings as “home,” “street,” or “office” [12] [13].
Classification, a machine learning method, assigns input patterns to predefined categories using a classifier. This process involves two main phases: training the classifier with samples representing each class and then categorizing unknown inputs into these classes. Different classification techniques use various algorithms and rules, which can impact accuracy based on the specific application [39]. Four notable classification algorithms for sound detection are:
1) Convolutional Neural Networks (CNNs): CNNs, commonly used in deep learning, excel in image classification but require a specialized approach for sound. Sound is converted into spectrogram images, which CNNs can then analyze. CNNs consist of layers that process sound data by generating feature maps to identify patterns [40] [41]. They are effective but require longer training times [8]. For example, CNNs have been used to classify bird sounds in normal and threatened conditions by analyzing spectrograms .
2) Deep Neural Networks (DNNs): Unlike standard neural networks with a single hidden layer, DNNs have multiple hidden layers, mimicking the human brain’s visual recognition model. This depth allows DNNs to achieve high precision by progressively recognizing complex information [29]. Research by Li et al. on DNN hyperparameters for speech recognition and audio analysis demonstrates their adaptability and high performance [8].
3) Recurrent Neural Networks (RNNs): RNNs are designed for sequence modeling, capturing temporal dependencies by considering both current input and previous hidden states. This allows RNNs to handle sequences of varying lengths and contexts. Arsenali et al. utilized RNNs for sound event classification, achieving high accuracy and sensitivity with their optimized model [42] [43].
4) Decision Trees: Decision trees, a non-linear classification method, use a hierarchical structure to eliminate classes sequentially until the correct class is reached. They are efficient for problems with many classes. The Ordinary Binary Decision Tree is a common variant, and research by Saifan et al. explored its application in sound engine classification [39].
Based on the different stages of sound detection technology, the entire process can be categorized and summarized into three primary stages: Preprocessing, Feature Extraction, and Model Training and Classification. These stages encompass the core methodologies and techniques employed in sound detection systems. Table 1 provides a detailed overview of these processes, highlighting the key steps, methods, and relevant research associated with each stage. This structured approach allows for a clearer understanding of how sound detection technology operates and the advancements made at each stage.
Table 1. Summary of key processes in sound detection technology.
Process Stage |
Description |
Key References |
Key Techniques/Methods |
Preprocessing |
Enhances audio data quality before feature extraction. |
[12]-[28] |
Noise Reduction, Equalization, Filtering, Silence Removal, Segmentation, Feature Windowing |
Feature Extraction |
Converts audio data into a numerical representation (feature vector) that captures key acoustic characteristics. |
[12] [13] [29]-[38] |
MFCCs, Log-Mel Energies, Spectrograms, Time-Frequency Analysis |
Model Training & Classification |
Develops models to classify audio recordings into predefined classes based on extracted features. |
[12] [13] [39]-[43] |
CNNs, DNNs, RNNs, Decision Trees |
The table categorizes sound detection technology into three main stages, highlighting key processes and methods employed in each stage, alongside relevant research, to present a clear overview of advancements in sound detection technology
4.2. Real-Time Sound Detection
Real-time processing refers to the immediate handling of data as it becomes available, with two main requirements: the processing must be completed faster than the data’s duration, and delays should be minimized to ideally zero. In computer operating systems, “real-time” also involves precise scheduling to handle events within a set timeframe [44].
Signal classification can be divided into three categories:
1) Analog Signal Processing: Deals with signals that have not been digitized, such as those from radios or older televisions.
2) Continuous Signal Processing: Focuses on signals with continuous amplitude variations, including modeling continuous systems, system function adjustments, and time-based filtering.
3) Discrete Signal Processing: Handles signals sampled and quantized at specific intervals, represented as a sequence of numbers. Discrete-time signals are crucial for real-time applications like sound event detection and speech recognition due to their ability to capture and process temporal dynamics [45].
Non-real-time signal processing involves manipulating pre-gathered and digitized signals without real-time constraints, while real-time processing demands precise timing from both hardware and software [46]. Key strategies for real-time sound detection include:
1) On-line Processing: Manages live data streams, computing results as the data is recorded or transmitted, typically using short buffers.
2) Incremental Processing: Optimizes on-line processing by reducing the delay between data input and analytical results.
Digital Signal Processing (DSP) significantly enhances real-time sound detection by digitally representing and analyzing signals. DSP involves breaking down signals, applying mathematical operations like filtering and Fourier transforms, and integrating results for effective analysis. Recent advancements in DSP technology have enabled real-time applications where analog methods are impractical [46].
An example of DSP application is in real-time arrhythmia classification, involving three stages: preprocessing to reduce noise, feature extraction using techniques like Wavelet Transform, and classification with algorithms such as probabilistic neural networks. DSP is essential in minimizing noise, extracting features, and classifying arrhythmias [47].
4.3. Security Aspect
Machine learning (ML) is becoming an essential part of modern cybersecurity, helping systems detect threats, analyze behavior, and respond automatically. However, as ML becomes more deeply integrated into security infrastructure, it also introduces new vulnerabilities. These risks largely stem from ML’s heavy reliance on large datasets and complex algorithms, which can compromise data confidentiality, integrity, and availability if not properly secured [48].
To mitigate these vulnerabilities—especially in the face of quantum computing threats—post-quantum cryptographic (PQC) schemes such as Kyber and NTRU have gained attention. These lattice-based algorithms offer quantum-resistant key encapsulation mechanisms, ensuring secure communication and data protection even under quantum-enabled attacks [49].
Kyber, in particular, can be embedded into ML pipelines to secure the transmission and storage of sensitive training data, model parameters, and updates. It protects against unauthorized access during remote deployment and communication, making it a strong candidate for ML-based systems operating in distributed environments [50].
Similarly, NTRU complements Kyber in offering low-latency encryption suitable for IoT and edge devices, which often rely on ML for local inference and security decisions. Both schemes have demonstrated resistance to side-channel attacks and ML-based cryptanalysis [49].
Many PQC schemes—including Kyber and NTRU—depend on efficient Number Theoretic Transform (NTT) implementations to accelerate polynomial multiplication, a core operation in lattice-based cryptography. High-performance NTT architectures such as pipelined R2MDC improve encryption and decryption speeds, enabling real-time operation in ML-driven security systems, especially those deployed in resource-constrained environments. Efficient NTT integration ensures that cryptographic operations do not become performance bottlenecks in ML workflows, allowing seamless real-time key exchange and digital signature generation—key elements in secure ML inference engines and autonomous systems [51].
Beyond encryption, ML can enhance system resilience by enabling adaptive responses to hardware faults or attacks. ML models can monitor cryptographic systems, analyze error patterns, and automatically trigger mitigation protocols. This approach is particularly effective for securing lightweight block ciphers like LED and HIGHT, which are often deployed in embedded systems with limited computational capacity. These adaptive techniques allow for the development of self-healing security architectures, which maintain cryptographic integrity despite environmental faults or malicious interference [52].
The importance of adopting quantum-resistant encryption was emphasized during the 2023 NIST Post-Quantum Cryptography Standardization process. Algorithms such as ML-KEM (based on Kyber) and ML-DSA (based on Dilithium) were selected as the future standards for secure communication. These algorithms are now being integrated into a wide range of ML-based systems, including smart devices, autonomous vehicles, and cloud services. As machine learning becomes more central to automation and critical infrastructure, combining it with PQC is key to building systems that are not only intelligent, but also secure and future-proof [53].
4.4. Challenges and Limitations
Creating automatic systems for sound event detection is a complex task with several challenges, particularly related to sound characteristics, data collection, and annotation. Addressing these challenges is crucial for the effectiveness of machine learning techniques [13]. Recent advancements offer potential solutions to these issues, which are detailed below.
A major challenge in sound event detection is handling overlapping sound events, a task known as polyphonic sound event detection. This involves identifying all coinciding sounds simultaneously [54]. To tackle this, supervised classification methods like RNNs [55] [56] and CNNs [57] are commonly used. These methods predict the presence of each sound event on a frame-by-frame basis, helping to manage the complexity of real-world sound environments.
Data challenges, such as insufficient samples and class imbalance, also impact classifier performance. Data augmentation, which involves synthetically increasing the data through techniques like pitch shifting, noise removal, compression, and time stretching, is a key strategy to improve the system’s performance by enhancing data representation [58].
Traditional audio processing often separates feature representation from classifier design, which can lead to suboptimal features. Deep Neural Networks (DNNs) address this by integrating feature extraction and classification, optimizing both processes simultaneously. For example, in speech recognition, lower DNN layers adapt to speaker characteristics while upper layers focus on class discrimination [42].
Another challenge is weakly labeled data, where only the presence or absence of events is known, but not their exact timings. Multiple Instance Learning (MIL) is a useful approach for dealing with such data. MIL treats the entire audio clip as a “bag” for classification when annotations are partial, allowing for effective sound event detection despite weak labels [59] [60].
As machine learning becomes increasingly integrated into critical sectors such as healthcare, finance, and cybersecurity, new security concerns and vulnerabilities have emerged. This research [48] highlights several key challenges that affect the secure deployment and operation of ML systems:
1) Emerging Vulnerabilities and Attack Surfaces: the widespread use of ML has introduced new entry points for malicious attacks. These vulnerabilities can be exploited to disrupt the integrity, availability, or confidentiality of ML systems. For example, attackers may take advantage of flaws in data preprocessing or the model architecture to alter outputs or extract sensitive training information.
2) Privacy vs. Accuracy Trade-offs: a core dilemma in ML security lies in balancing data privacy with model performance. Techniques such as differential privacy aim to protect individual data by introducing noise, but this often comes at the cost of reduced accuracy. As the number of queries increases, the risk of privacy leaks or performance degradation also grows.
3) Confidentiality and Intellectual Property Concerns: in applications involving sensitive or proprietary information, such as patient records or financial data, maintaining the confidentiality of ML models and their underlying data is critical. If an attacker gains access to model parameters, they may be able to reconstruct proprietary algorithms or extract confidential data, resulting in serious privacy breaches and intellectual property theft.
The growing deployment of ML also exposes it to a variety of sophisticated attacks, as discussed in [52]:
1) Adversarial Attacks: these attacks involve subtly altering input data to fool the model into making incorrect predictions. Such manipulations can undermine the reliability of ML-based security systems by causing false positives or negatives.
2) Model Inversion Attacks: in this case, attackers analyze model outputs to reconstruct the original training data, potentially revealing personal or confidential information that was assumed to be protected.
3) Data Poisoning Attacks: by injecting malicious or misleading data into the training set, attackers can corrupt the learning process. This compromises the integrity of the model and can be exploited to influence its behavior in harmful ways.
4) Evasion Attacks: these attacks involve modifying malicious behavior or input data in a way that avoids detection by ML-based security systems. For example, attackers might disguise malware traffic to bypass an intrusion detection system trained on known patterns.
5) Denial of Service (DoS) Attacks: ML systems can also be overwhelmed by large volumes of requests or data, causing them to slow down or crash. In critical environments, such as automated surveillance or fraud detection, such disruptions can have severe consequences.
5. Speech Recognition
Recent advancements in speech recognition have enhanced communication across languages and interactions with various devices. These improvements stem from three main factors: increasing computational power from multi-core processors and GPU clusters, access to extensive datasets, and the rise of mobile, wearable, and smart home technologies [61].
Speech technology impacts both human-to-human (HHC) and human-to-machine communication (HMC). In HHC, research has focused on assisting individuals with speech impairments through applications like Google’s Speech API, which converts speech to text with high accuracy [62]. Studies also address transcription for the deaf and hard of hearing and explore deep learning models for language translation [63] [64].
In HMC, advancements include voice-activated smart home systems, though these often overlook less common languages like Romanian. To address this, new acoustic and grammar models for Romanian have been developed, alongside remote speaker recognition techniques [65] [66]. Additionally, the field covers voice search and interaction with mobile devices and entertainment systems [67].
This paper emphasizes automatic speech recognition as a critical component in spoken language technology, focusing on its role in real-time audio transcription.
Automatic Speech Recognition
In this book [61], the architecture of automatic speech recognition (ASR) is elucidated, comprising four primary components: Signal processing and feature extraction, acoustic model (AM), language model (LM), and hypothesis search. Signal processing readies audio input by converting it into feature vectors. The AM evaluates the likelihood of feature sequences, while the LM estimates word sequence probabilities. The hypothesis search combines AM and LM scores to yield the recognition result. The AM must address challenges such as variable-length feature vectors and acoustic variability, stemming from factors like speaker characteristics, speech style, noise, and accents. Real-world ASR systems encounter further complexities, including extensive vocabularies, spontaneous speech, and multilingual contexts. While traditional ASR systems employed features like MFCC and RASTA-PLP with GMM-HMM models trained via maximum likelihood criteria, recent advances include discriminative training methods such as MCE and MPE in the 2000s and the adoption of DNNs and discriminative hierarchical models like CD-DNN-HMM, substantially improving accuracy due to enhanced computational capabilities and more extensive training data.
6. Related Applications for Deaf and Hard-of-Hearing Individuals
This section will present research papers and applications aimed at improving the lives of deaf and hard-of-hearing individuals. Numerous studies focus on leveraging recent technologies, such as sound detection and speech recognition, to enhance their safety and overall quality of life.
6.1. Related Research Papers
The Enssat application [68] utilizes Google Glass as a wearable device to support individuals who are deaf or hard of hearing in both Arabic and English. This application primarily focuses on functions such as sound detection and speech recognition. The sound detection feature of Enssat utilizes the microphone of either a mobile phone or Google Glass to identify ambient sounds. When a sound is detected, a snippet of it is recorded and sent to another thread for identification. The system then compares this recorded sound with a set of stored sounds using the MusicG library to determine the degree of similarity. Regarding speech recognition, Enssat offers real-time transcription of speech. It employs Google’s Speech-to-Text service to convert audio files into text. The challenge lies in effectively processing continuous speech and segmenting it into snippets for accurate transcription. Additionally, the Enssat application provides translation capabilities, enabling the translation of both spoken words and text captured in images. To achieve this, the system utilizes Google’s Translation API to deliver real-time translation services.
A different document explores the development of a specialized virtual assistant catering to individuals with hearing impairments, highlighting its proficiency in identifying and categorizing various sounds. The proposed remedy encompasses a sound classification module, a gesture recognition module, and a multilingual translation module. The sound classification module is engineered to recognize and categorize diverse sounds, such as those produced by vehicles, to notify users of potential hazards. It leverages audio data from the UrbanSound8K dataset and employs a deep neural network for sound recognition. The gesture recognition module translates gestures from Indian Sign Language into text and audio, facilitating communication between non-deaf individuals and those with hearing impairments. The multilingual translation module converts the generated text into various regional languages, offering translation services for hearing-impaired individuals in India. This solution is seamlessly integrated into an Android application and has undergone a comparative analysis with existing apps, assessing factors such as response time, accuracy, output predictions, and alert systems [6].
Saifan presents the Deaf Assistant Digital System [39], a solution utilizing smartphones to provide alerts for individuals with hearing impairments across various situations. Employing vibrations and visual notifications on the smartphone screen, the system ensures effective communication with the user. The paper delves into the technical intricacies of the speech and sound recognition engines, detailing the utilization of deep auto-encoder-based low-dimensional feature extraction from FFT spectral envelopes. This approach enables the identification of diverse sounds and words. Additionally, the paper highlights the application of Praat, a computer program for speech analysis, synthesis, and manipulation, in extracting sound features. The system’s matching engines play a crucial role in comparing recognized words and sounds with predefined cautionary words and sound alerts. Upon identifying a match, these engines trigger appropriate actions such as vibration and visual effects to alert individuals with hearing impairments.
In this paper [69], the significance of sound detection is examined, and diverse technologies and systems designed for this objective are explored. The paper emphasizes the growing presence of comprehensive sound detection systems in the market. In the realm of sound detection technology, a proof-of-concept for a sound detection algorithm based on Gaussian Mixture Model (GMM) is discussed. Additionally, the paper briefly touches upon the application of Gaussian Mixture Models for speaker identification and verification in the context of speech recognition.
The iHelp application [70] introduces a real-time mobile emergency assistance system designed to aid deaf-mute individuals or elderly individuals living alone in promptly and effectively reporting emergencies. This system employs mobile application software installed on smartphones, enabling users to report emergencies through SMS, even in the absence of internet access. By doing so, it optimizes the dispatching of rescue units and enhances the overall success rate of emergency rescue operations. The system is comprised of three key components: the report subsystem, dispatch system, and rescue subsystem.
6.2. Related Mobile Applications
The Sound Alert App functions as a tool for capturing and informing users of significant environmental and household occurrences. It can identify various sounds like doorbells, phone rings, microwave beeps, alarms, and intercoms without the need for pre-recording. What distinguishes this solution is its seamless integration with existing building infrastructure and alarm systems, providing a cost-effective alternative to flashy lights and expensive hardware setups. This app proves especially beneficial for individuals who are hard-of-hearing, deaf, elderly, or heavy sleepers. By activating “Detection Mode,” the app’s intelligent algorithm continuously monitors the environment through the smartphone’s microphone. Additionally, it can sync with Pebble Watch for extra notification options, including vibration, flashing lights, and on-screen icons with event names [71]. Another similar app, The Deaf and Hearing Impaired (APK), is designed to assist deaf or hearing-impaired individuals by alerting them through vibration and flashlight signals when a loud sound occurs nearby [72].
The Android app Live Transcribe & Sound Notifications enhances accessibility for individuals with hearing impairments. It enables real-time transcription of spoken words in over 80 languages and dialects, allowing users to customize word additions. The app also notifies users of various sounds, including potentially risky situations. Users have the flexibility to adjust settings, save transcriptions for three days, and search within saved transcriptions. Developed in collaboration with Gallaudet University, a leading institution for the deaf and hard of hearing, the app is compatible with Android 6.0 and newer devices. It incorporates features such as vibrating when the user’s name is spoken and supports external microphones for improved audio reception [73].
Rogervoice, a revolutionary call transcription application, has transformed phone communication for individuals who are deaf or hard of hearing. By offering real-time call subtitles in over 80 languages, it enables users to independently connect with family, friends, medical professionals, and customer service helplines. The app is designed to be user-friendly, allowing calls to be initiated either from contacts or by entering numbers manually. Conversations are transcribed instantly, and users have the option to respond through speech or typing, with a voice synthesizer delivering text messages. It’s important to note that Rogervoice does not support emergency calls or premium-rate numbers. Subscriptions are required for calls to individuals who do not use the application, and pricing details can be found on the website [74].
In the pursuit of modernizing communication within the realm of security and improving the efficiency of security personnel in managing emergency reports, the General Directorate of Public Security has introduced the “Kulluna Amn” mobile application. This application is designed to actively involve citizens and residents in the security framework and has been launched with the direct support and guidance of HRH Prince Mohammed bin Nayef, Deputy Prime Minister and Minister of Interior. The app empowers users to report unusual incidents, which are then transmitted to the thirty-nine operation rooms situated across the kingdom. Users can furnish details about incidents, including photos and GPS location, choose the incident category, and even pinpoint the nearest police or traffic department based on their current geographical coordinates. “Kulluna Amn” signifies a noteworthy stride in enhancing emergency response and encouraging public participation in upholding security [75].
The TapSOS application serves as a crucial tool for reaching Emergency Services in situations where verbal communication is challenging or unsafe, especially for individuals with hearing impairments. Users establish profiles containing essential information, which is communicated to Emergency Call Handlers through visual icons. A medical profile provides valuable information for First Responders. The GPS feature automatically identifies the user’s location, allowing manual adjustments for precision. By responding to a series of questions aligned with Emergency Services protocols, users trigger alerts that are directly transmitted to the UK’s 999 Emergency Call Handlers. This makes TapSOS an indispensable tool for non-verbal emergency communication [76].
6.3. Comprehensive Comparsion
This section explores a variety of research papers and mobile applications designed to improve the lives of deaf and hard-of-hearing individuals. These solutions leverage technologies such as sound detection, speech recognition, and real-time transcription to provide timely alerts, enhance communication, and increase situational awareness. Applications like Sound Alert App [71], Live Transcribe & Sound Notifications [73], and Rogervoice [74] offer real-time features powered by cloud-based services, while others like iHelp [70] operate offline to ensure accessibility in emergencies. Solutions such as Saifan’s Deaf Assistant [39] and Enssat [68] integrate deep learning for accurate sound classification and multilingual support, including Arabic. Meanwhile, research initiatives like the Virtual Assistant with gesture recognition [6] and YAMNet-based firearm detection system [69] demonstrate the effectiveness of deep learning in both general and specialized sound classification tasks. Other apps, such as Deaf and Hearing Impaired (APK) [72], Kulluna Amn [75], and TapSOS [76], emphasize safety, emergency reporting, and user-friendly design for accessible communication. Collectively, these solutions highlight the diversity of assistive technologies, balancing real-time performance, offline functionality, localization, and usability to support users with hearing impairments. Table 2 presents a summary and comparison of these applications and studies. Table 3 classifies them based on core assistive features such as sound detection, live transcription, Arabic support, emergency services, user interface design, and alert notifications.
Table 2. Summary of related applications and research papers.
Category |
Application/Research Paper |
Description |
Reference |
Related Research Papers |
Enssat Application |
Utilizes Google Glass for sound detection and speech recognition, offers real-time transcription and translation. |
[68] |
Specialized Virtual Assistant |
Features sound classification, gesture recognition, and multilingual translation for individuals with hearing impairments. |
[6] |
Deaf Assistant Digital System |
Uses smartphones to provide alerts via vibrations and visual notifications, with sound and speech recognition. |
[39] |
Sound Detection Technology |
Discusses a proof-of-concept for sound detection using Gaussian Mixture Models (GMM). |
[69] |
|
iHelp Application |
Mobile emergency assistance system for deaf-mute or elderly individuals, includes reporting emergencies via SMS. |
[70] |
Related Mobile Applications |
Sound Alert App |
Identifies and alerts users to significant sounds, integrates with building infrastructure, and supports Pebble Watch. |
[71] |
The Deaf and Hearing Impaired (APK) |
Alerts users to loud sounds nearby through vibrations and flashlight signals. |
[72] |
Live Transcribe & Sound Notifications |
Provides real-time transcription of spoken words and notifications of various sounds in over 80 languages. |
[73] |
Rogervoice |
Real-time call transcription application with subtitles in over 80 languages, enabling phone communication. |
[74] |
Kulluna Amn |
Mobile app for reporting incidents to security forces, includes GPS location and incident details. |
[75] |
TapSOS |
Emergency communication app for non-verbal communication, provides visual icons and GPS location for emergency services. |
[76] |
aThis table reviews technologies and research designed to improve the lives of deaf and hard-of-hearing individuals, focusing on sound detection and speech recognition.
Table 3. Feature classification of applications and research.
Features |
Sound detection |
Live transcriptions during the call |
Support Arabic language interface |
Emergency call |
Easy to use & attractive interface |
Android Application |
Alert notification |
Enssat |
√ |
|
√ |
|
|
|
√ |
Virtual Assistant for Hearing Impaired |
√ |
|
|
|
√ |
√ |
√ |
Deaf Assistant Digital System |
√ |
|
|
|
|
√ |
√ |
Sound Detection Technology |
√ |
|
|
|
|
√ |
√ |
iHelp |
|
|
|
√ |
|
|
√ |
Sound Alert |
√ |
|
|
|
√ |
|
√ |
The Deaf and Hearing Impaired (APK) |
√ |
|
|
|
|
√ |
√ |
Google live transcribe |
|
|
√ |
√ |
√ |
√ |
√ |
Rogervoice |
|
√ |
|
|
√ |
√ |
|
Kulluna Amn |
|
|
√ |
√ |
√ |
√ |
|
TapSOS |
|
|
|
√ |
√ |
√ |
|
aThis table compares the related studies and applications, based on functionality and features.
7. Discussions
Based on the discussion presented in this review paper, it can be concluded that artificial intelligence plays a crucial role in enabling real-time alarm sound detection applications designed to support individuals who are deaf or hard of hearing and speak Arabic. Such applications can significantly facilitate communication with government agencies in Saudi Arabia during emergency situations. In other words, an effective approach would be to design an integrated application that combines key elements from existing solutions. The application should:
Incorporate Sound Detection Technology: Utilize algorithms similar to those in the Sound Alert App and The Deaf and Hearing-Impaired app to detect alarm sounds. This will enable the app to identify and alert users about important environmental and home events, specifically tailored to the needs of deaf and hard-of-hearing individuals.
Implement Speech Recognition and Transcription Features: Drawing from Live Transcribe & Sound Notifications and Rogervoice, integrate real-time transcription of spoken words in Arabic. This feature will aid in communication during emergency situations, enabling individuals to understand spoken information.
Facilitate Communication with Government Agencies: Inspired by Kulluna Amn & TapSOS, the application should allow users to communicate with the relevant authorities, specifically in Saudi Arabia. This ensures a direct link to the appropriate agencies during emergencies.
Combining these elements into a single application tailored to the Arabic-speaking population, particularly in Saudi Arabia, would address the core challenges of sound detection, communication during emergencies, and interaction with relevant government agencies for the deaf and hard-of-hearing community.
8. Conclusions
By utilizing breakthroughs in artificial intelligence and machine learning, modern assistive hearing technologies have made significant progress over conventional auditory aids. These modern systems have better accuracy, more contextual understanding, and better security measures. Today’s remedies, unlike earlier standalone devices, are connected, flexible, and significantly more sophisticated, signaling a radical change in the way hearing aid is provided.
In summary, incorporating advanced technologies especially in the areas of machine learning and artificial intelligence shows a lot of potentials to improve the safety and the quality of life of hearing-impaired individuals. This paper has analyzed assistive technologies, sound recognition, and sonification for the enhancement of the functionality of patients suffering from hearing loss. Hearing aids, cochlear implants, and mobile apps breakdown communication barriers and promote self-sufficiency, effectively changing the status quo of everyday routines. On the other hand, machine learning-centered technologies for sound detection provide advanced means of real-time situational awareness for safety, security, and environmental monitoring.
The key findings of this review highlight the transformative impact of these technologies. Augmented hearing aids and cochlear implants are essential; both assist in increasing overall auditory fulfillment, while empowering a sense of independence. Sound detection systems driven by machine learning based models, generate alerts in time and also provide contextual information when required that specifically addresses the safety problems encountered to people with hearing impairments. Further, the evolution of speech recognition technologies enables communication across disparate contexts enabling greater context-aware support for integration and cross-interaction within different environments.
There is still a lot of room for innovation by using cutting-edge ML approaches, even with these advancements. Future research should concentrate on a few key areas in order to overcome existing constraints and increase the functionality of assistive audio devices:
Models Based on Transformers: In order to improve the identification of complicated or overlapping audio events, especially in noisy or erratic situations, use architectures like the Audio Spectrogram Transformer (AST).
Federated Learning: To train models across several devices while preserving data security and user privacy, use decentralized learning frameworks.
Context-rich datasets: To enhance real-world applicability and generalization, curate and label a wide variety of audio datasets that contain uncommon and contextually relevant sound events.
Multimodal Integration: To facilitate more context-aware and reliable hazard or event identification, integrate audio signals with sensor-based or visual data streams.
Realtime Personalization: Create flexible algorithms that can adapt sound recognition models to individual user environments and preferences in real time.
Postquantum security includes integrating postquantum cryptographic algorithms, particularly those that were highlighted in the 2023 NIST competition, in order to safeguard sensitive user data and guarantee the integrity of communications between assistive systems.
By integrating these machine learning techniques, future assistive hearing devices can offer more individualized, safe, and dependable assistance, enabling people with hearing loss to live more independently and safely in their surroundings.