Impact of Image Enhancement on Deep Learning-Based Recognition of Activity Prompts in Children with Autism Using Motion History Images

Autism Spectrum Disorder (ASD) is a mental and neurological disease in children that affects social interaction, communication, and behavior [1]. The prevalence of ASD increases globally, from 1 in 54 children in 2016 to 1 in 44 in 2018, and in Indonesia from 1 in 500 in 1995 to 1 in 50 in 2013, affecting an estimated 2.4 million individuals [2]. Children with ASD may not be fully independent, but they need to be able to perform daily activities, such as eating and drinking [3]. However, they often refuse food, become overly selective, or chew too slowly. Meal planning and support are essential to help them with their nutrition intake [4].

Prompts play a crucial role in helping children with ASD follow structured routines such as eating and drinking, and their consistent use across various settings enhances the effectiveness of daily activity support. Prompts can be physical, gestural, or verbal [5]. Prompts—whether verbal (e.g., “drinking”) or gestural (e.g., pointing to a spoon)—serve as cues that guide the child through each step for a task. A child’s response to prompts, such as picking up a spoon or bringing a cup to the mouth, indicates their understanding, attention, and task execution, providing valuable insight into functional performance in daily life.

Technologies in this field include activity prompt recognition, where image processing methods such as Histogram Equalization (HE), Contrast Stretching (CS), and Contrast Limited Adaptive Histogram Equalization (CLAHE) enhance image quality and reduce noise [6] and improve Convolutional Neural Network (CNN) performance [7].

Deep learning is effective for classification tasks due to its adaptable architecture and ability to automatically extract meaningful features from raw data [8]. The architecture of CNN can identify patterns within images by leveraging convolutional, pooling, and activation layers—collectively known as convolutional features [9].

To complement CNNs in recognizing motion information, Motion History Image (MHI), which is known for its computational efficiency and low memory usage [10]. MHI converts video sequences into static images. It is effective for motion representation as it can highlight recent object movements while minimizing background changes. In CNN, MHI is a preprocessing step for motion recognition [11]. Numerous studies have employed MHI for action and activity recognition. MHI and Support Vector Machines (SVM) were used in a study by Tsai et al. [12] to recognize human actions. MHI and CNN were used in a study by Ahn et al. [13] to recognize cow actions. A study by Núñez et al. [14] integrated MHI with deep learning techniques to recognize daily activities, while Sahoo et al. [15] employed historical images along with CNN for activity recognition.

Previous studies [6, 16-19] employed CNN to identify ASD. The study by Singh et al. [16] demonstrated that CNN can be used to diagnose ASD from children’s video recordings, achieving an accuracy of 85%. However, the study did not consider the impact of image or video quality on the model’s performance, even though the dataset was collected from highly diverse YouTube sources with significant variations in lighting, noise, and resolution. Meanwhile, Haweel et al. [17] identified ASD using a CNN algorithm using brain image datasets, distinguishing between children with ASD and typically developing youngsters, and reached an 78% accuracy. However, this method remains limited to neuroimaging data characterized by low temporal resolution, high cost, and dependence on clinical facilities. Moreover, the proposed model primarily focuses on brain frequency features and does not incorporate directly observable behavioral aspects relevant to ASD detection. Furthermore, Sherkatghanad et al. [18] utilized open-access brain imaging datasets, and the experiments demonstrated that the CNN-based model achieved an accuracy of 70.22% in classifying individuals with ASD. Next, Huang [19] employed fMRI data and an SVM technique, attaining an accuracy of 70% in ASD classification. Both studies [18, 19] focused on diagnosing ASD in children using deep learning or machine learning approaches; however, these studies did not include feature extraction processes and were limited to neuroimaging data without considering behavioral aspects of ASD. In addition, previous work [6] on identifying daily activities in children with ASD included six drinking and eight eating sequences, achieving an accuracy of 85%. Although the performance was satisfactory, the study did not incorporate feature extraction, resulting in high computational complexity. This limitation makes such an approach less suitable for real-time applications that require efficient processing.

This study extends the work presented in reference [6] by focusing on detecting prompts within daily activities and integrating image enhancement with motion-based feature extraction using MHI within CNN architectures. Unlike previous studies that utilized uncontrolled online video sources, this work employs real-world activity videos recorded by therapists and parents under guided observation to ensure data reliability. The video was processed using three video enhancement techniques—HE, CS, and CLAHE—as applied in the reference [6]. Enhancement results were quantitatively evaluated to assess image quality improvement, and the best-performing method was integrated with MHI-based motion features for temporal representation before CNN-based prompt recognition. This integration addresses the visual variability and motion inconsistency commonly found in real-world ASD datasets, improving both recognition robustness and computational efficiency.

The proposed study introduces a novel integration of image enhancement and MHI within CNN architectures, namely VGG19 and MobileNetV2. This study contributes to literature in three key ways. First, the study employs real-world video performed by children with ASD, recorded by therapists and parents in natural, uncontrolled environments. Second, the study investigates image enhancement techniques to improve the recognition accuracy of these prompts under challenging visual conditions. Third, it proposes a method for classifying prompt types by applying motion-based feature extraction using MHI and image enhancement techniques, followed by activity recognition using a CNN across multiple datasets with enhanced image contrast. Together, these contributions highlight the study’s novelty in bridging temporal and visual domains through image enhancement and motion representation for robust and efficient prompt recognition in children with ASD.

This part describes the experimental findings of recognition (integration of CLAHE and MHI with CNN), compares them to state-of-the-art methods, and discusses.

3.1 Video enhancement evaluation

Video enhancement evaluation for eating activities is displayed in Tables 6-8. The average MSE for HE is 98.702, CS is 91.937, and CLAHE is 10.246. The average PSNR values are 17.431 for HE, 11.302 for CS, and 38.391 for CLAHE. The average SSIM is 0.860 for HE, 0.801 for CS, and 0.997 for CLAHE.

Table 6. The average MSE of eating activities

Table 7. The average PSNR of eating activities

Table 8. The average SSIM of eating activities

Table 9. The average MSE of drinking activities

Table 10. The average PSNR of drinking activities

Table 11. The average SSIM of drinking activities

Tables 9-11 display video enhancement evaluation for drinking activities. The average MSE result is 97.667 for HE, 93.693 for CS, and 10.586 for CLAHE. The average PSNR for HE is 17.024, CS is 11.257, and CLAHE is 38.275, while the average SSIM values are 0.843 for HE, 0.796 for CS, and 0.997 for CLAHE.

3.2 Extracting features

The feature extraction process was applied to a sample video from the eating activity dataset, specifically "Verbal (235).mp4," which consists of 124 extracted frames, as illustrated in Figure 6. A MHI representation is shown for frames 15 to 30. The frames were first enhanced using CLAHE, as depicted in Figure 6(a). Object segmentation was performed using Otsu thresholding, with the result shown in Figure 6(b). Subsequently, an MEI was generated in Figure 6(c), followed by the final MHI representation shown in Figure 6(d).

6-1.png

(a) CLAHE

6-2.png

(b) Object segmentation

6-3.png

(c) MEI

6-4.png

(d) MHI

Figure 6. Result of feature extraction

3.3 Evaluation of activity prompts recognition

This study applies integration of CLAHE and MHI with VGG19 or MobileNetV2 to enhance recognition activity prompts, specifically for eating and drinking, using evaluation metrics and AUC-ROC.

Table 12 presents the recognition result of the proposed approach. Proposed approach 1 achieved the highest accuracy of 71.31%, while proposed approach 4 yielded the lowest accuracy at 68.85%. Meanwhile, proposed approach 5 obtained the best performance with an accuracy of 73.96%, whereas proposed approach 8 recorded the lowest accuracy at 67.71%.

Table 12. Recognition results of the proposed approach for eating and drinking activities

Table 13. Computation time of the proposed method

Table 13 presents the computation time of the proposed method. Proposed approach 1 achieved the highest accuracy with a computation time of 543 seconds, while proposed approach 4 yielded the lowest accuracy with a computation time of 596 seconds. Meanwhile, proposed approach 5 obtained the best performance with a computation time of 308 seconds, whereas proposed approach 8 recorded the lowest accuracy with a computation time of 452 seconds.

Based on Tables 12 and 13, MHI and (VGG19 and MobilenetV2) with enhancement (CLAHE) yield higher accuracy than MHI and (VGG-9 and MobileNetV2) without enhancement (grayscale) models. The use of CLAHE and MHI with (VGG19 and MobileNetV2) improves accuracy, indicating that CLAHE enhances the visibility of motion features in MHI, enabling more effective pattern recognition, while MHI and CLAHE reduce computation time.

Table 14 presents the AUC-ROC results of the eight proposed methods. The average AUC-ROC for the physical prompt is 0.82, for the gesture prompt 0.65, and the verbal prompt 0.75. The physical prompt has the highest average score, while the gesture prompt has the lowest. The fourth method achieves the highest AUC-ROC (0.94) for physical prompts, indicating strong classification ability with minimal misclassification risk. In contrast, the eighth method has the lowest AUC-ROC (0.60) for gesture prompts, showing difficulty in gesture classification.

Table 14. Results of AUC-ROC

3.4 Comparison of proposed approaches

The suggested approach's conclusions are evaluated by comparing them to current approaches. Table 15 provides a comparison of the recommended techniques.

Table 15. Comparison of proposed approaches

Table 15 provides a comparative analysis of this study and previous research, emphasizing the impact of image enhancement on model performance. Previous studies [12-15] have shown that MHI combined with SVM achieved accuracies ranging from 67% to 72%, while CNN-based approaches for activity recognition reported accuracies between 63% and 70%. Nonetheless, the two methods have limitations in terms of precision and computational efficiency, with computation times in some situations exceeding 900 seconds. These findings underscore the need for approaches that improve both recognition performance and computational efficiency. In addition, several studies [16-19] investigated ASD diagnosis using secondary data obtained from public repositories. While most existing studies on ASD activity recognition report accuracy as the primary evaluation metric, they rarely address computational efficiency. In contrast, our study emphasizes real-time feasibility by reducing computation time while maintaining competitive accuracy.

Regarding comparison with prior studies in Table 15, several referenced works did not report computation time, limiting direct efficiency comparison. To ensure transparency, the current study explicitly reports computation time for each tested configuration, demonstrating that CLAHE integration not only improves accuracy but also reduces processing time, indicating practical gains in real-time applicability.

3.5 Discussion

Tables 6-11 reveal that CLAHE proves to be effective, as it achieves an average of 10.246 for MSE, 38.391 for PSNR, and 0.997 for SSIM in the eating dataset, as well as an average of 10.586 for MSE, 38.275 for PSNR, and 0.997 for SSIM in the drinking dataset. An MSE value near 0 and <30 indicates good results and reduced error. PSNR reflects image processing quality, with values >35 dB indicating high accuracy and <35 dB, suggesting otherwise. SSIM ranges from -1 to 1, where 1 signifies identical images, and a value near zero or negative reflects negligible similarity [33]. Considering these results, CLAHE was chosen as the image enhancement method for recognition.

Tables 12 and 13 show that CLAHE preprocessing improves classification performance compared to grayscale, increasing accuracy from 69.67% to 71.31% for eating activities and from 72.92% to 73.96% for drinking activities. This improvement is attributed to CLAHE’s ability to enhance local contrast, which is particularly useful under varied lighting and background conditions common in real-world ASD recordings. Moreover, CLAHE integration reduced computation time, from 648 to 543 seconds for eating activities and from 420 to 308 seconds for drinking activities, demonstrating its dual benefit of improving accuracy and efficiency [34]. Although the numerical improvements in accuracy are modest (around 1–2%), the consistent gains across all metrics and the substantial reduction in computation time demonstrate the practical advantage of integrating CLAHE and MHI with CNN for real-time recognition tasks. These results indicate that between contrast optimization (CLAHE) and temporal motion encoding (MHI) enhances not only improve image quality but also facilitate faster convergence and more efficient feature extraction in CNN-based models.

In this study, each configuration was trained and tested multiple times, and the best-performing result was reported to reflect the model’s optimal capability under each enhancement setting. Formal statistical significance testing (e.g., p-value computation) was not conducted because the evaluation focused on best-performing results rather than averaged outcomes across repeated runs. Future work will include statistical validation through multiple-run averaging to strengthen empirical reliability.

Table 14 presents the AUC-ROC results of the proposed approach, with the physical prompt achieving a score of 0.89, indicating strong classification performance. This suggests the model effectively distinguishes classes with minimal errors [31]. The AUC-ROC for the gesture prompt is the lowest at 0.75, indicating adequate performance but a higher likelihood of errors [32]. Gesture prompting is the least effective due to its ambiguity compared to physical and verbal methods. Physical prompts offer clear, direct cues, while verbal prompts provide explicit linguistic meaning, which is easier to interpret. Additionally, labeling gesture data is challenging, as it requires visual movement interpretation, increasing the risk of errors that can impact model accuracy.

Table 15 shows that applying video enhancement improves CNN classification accuracy. For the eating activities dataset, the MHI–VGG19 model with CLAHE achieves 71.31%, compared to 69.67% without enhancement (grayscale). For drinking activities, the same model reaches 73.96% with CLAHE, compared to 72.92% without enhancement. This improvement can be attributed to CLAHE, which increases image contrast and clarity, thereby enabling CNN to extract more discriminative features for classification. In addition, VGG19 demonstrates higher accuracy than MobileNetV2. Specifically, for eating activities, MHI–VGG19 with CLAHE achieves 71.31% while MHI–MobileNetV2 with CLAHE achieves 70.49%. For drinking activities, MHI–VGG19 with CLAHE achieves 73.96% compared to 69.79% for MHI–MobileNetV2. The superior performance of VGG19 can be explained by its deeper 19-layer architecture [27], which allows the extraction of more complex features and better detection of subtle motion patterns in MHI.

The relatively low classification accuracy in this study is primarily due to the dataset’s inherent complexity and variability, like the Real-Time Cow Action [13] and HMDB51 [15] datasets, the data used in our experiments exhibit substantial variations in motion patterns, camera angles, and background clutter, which pose significant challenges for action recognition models. In addition, many gestures exhibited by children with autism tend to be subtle, non-standardized, and vary significantly across individuals, making them difficult to interpret—even for trained human observers [35]. These factors collectively represent a major challenge in developing robust image-based activity recognition systems for this population.

These characteristics restrict the model's ability to generalize across samples, lowering overall performance despite the robustness of the proposed strategy. Furthermore, the usage of MHI as the dominant temporal representation may add to the reduced accuracy, as MHI tends to oversimplify motion dynamics in complex scenes with overlapping actions, background motion, or occlusions. This limitation makes it difficult to capture fine-grained temporal cues required for distinguishing subtle action differences in highly variable datasets. Similar challenges in applying MHI to complex datasets have also been reported in previous studies, where the method showed limited effectiveness in capturing temporal nuances in unconstrained video scenarios [13, 15].

Table 15 shows that the accuracy achieved in this study is lower than that reported in the study [6]. However, it reduces the computational time of from 39.448 seconds to 308 seconds. Notably, this shorter processing time corresponds to the highest accuracy obtained in our experiments, reaching 73.96%. This substantial improvement in computational efficiency enhances the system’s feasibility for real-time detection scenarios, which is crucial in practical applications. Fast processing time directly contributes to better scalability [36], enabling the system to provide prompt responses during daily routines for children with ASD. By reducing the number of processed frames to one per video using MHI, the computational cost is minimized. Although this results in a loss of temporal information and may have a modest impact on classification accuracy, the trade-off is acceptable in the real world, resource-constrained environments where real-time interaction is essential.

Enhancing MHI with preprocessing methods (e.g., Gamma Correction [37], Gaussian Blur [38] can preserve motion patterns, while leveraging diverse datasets, pose estimation, and optical flow may reduce gesture ambiguity. Integrating CNN and RNN is expected to further improve recognition accuracy, reliability, and overall system performance.