Impact of Combining RGB and Grayscale Images on Hotspot Detection in Solar Panels Using Inception Resnet V2 Architecture

This research also explores various new approaches in image processing, including the use of hybrid methods and multispectral imaging to enhance the detection and classification of damage in solar panels [28]. In this context, architectures like Inception ResNet V2 not only offer efficient solutions in terms of computational cost but also provide high accuracy, making them an ideal choice for applications that require high-resolution image processing [29, 30]. Through this approach, this study aims to achieve significant improvements in the early detection of hotspots, ultimately supporting global efforts to maximize the utilization of solar energy [31].

In Inception ResNet, each block consists of several parallel convolutional paths that are merged into a single output and then combined with a residual connection.

$A^{(l+n)}=\operatorname{ReLU}\left(\sum_i\left(W^{(l, i)} * A^{(l-1)}\right)+b^{(l)}+A^{(l)}\right.$  (1)

To reduce spatial dimensions while increasing or maintaining the number of filters [32]. Inception ResNet architecture uses a dimension reduction strategy to decrease spatial dimensions while increasing or maintaining the number of filters. This strategy involves using pooling layers (such as max-pooling or average-pooling) or convolutions with a stride greater than 1. This process can be formulated as:

$A^{(l)}=$ Pooling $\left(A^{(l-1)}\right)$   (2)

or

$A^{(l)}=W^{(l)} * A^{(l-1)}+b^{(l)}$ (3)

where, Pooling $\left(A^{(l-1)}\right)$ is a pooling operation that reduces the spatial dimensions of ($A^{(l-1)}$) while maintaining or even increasing the number of filters as needed for the subsequent layers. This is crucial for maintaining computational efficiency and avoiding overfitting, especially in very deep networks like Inception ResNet [33].

Furthermore, to control the output size generated by each block, various regularization techniques are employed, such as dropout and batch normalization. Batch normalization can be formulated as:

$\hat{Z}^{(l)}=\frac{Z^{(l)}-\mu^{(l)}}{\sqrt{\left(\sigma^{(1)}\right)^2+\epsilon}}$   (4)

$A^{(l)}=\operatorname{ReLU}\left(\hat{Z}^{(l)}\right)$    (5)

where, $\hat{Z}^{(l)}$ is the normalized output, $\mu^{(l)}$ is the mini-batch mean of the output $\hat{Z}^{(l)}, \sigma$ is the mini-batch standard deviation, and $\epsilon$ is a small value to prevent division by zero [34]. To optimize the entire network, a commonly used loss function for classification tasks is categorical cross-entropy, which is formulated as:

$L=-\sum_{i=1}^N y_i \log \left(\hat{y}_i\right)$      (6)

where, $y_i$ is the labelling, and $\hat{y}_i$ is the probability predicted by the model for class $i$. This loss function measures the difference between the predicted probability distribution of the model and the actual distribution, and it is used to update the model parameters through backpropagation [35]. Overall, the Inception ResNet architecture combines the flexibility of Inception blocks with the stability of residual connections, resulting in a robust model capable of handling complex image classification tasks [34].

Table 3. Image augmentation

As a result of the image augmentation process, the dataset size increases significantly (excluding the validation split). Augmentation techniques applied to the dataset introduce new variations of the existing images, enhancing the model's ability to generalize across different conditions and scenarios. These transformations include changes in orientation, scale, brightness, and positioning, which simulate real-world variability in data. The details of the augmentation techniques applied are summarized in Table 3, providing an overview of the strategies implemented to diversify the dataset and strengthen the model’s robustness.

Below is a summary of the outcomes produced by the augmentation process. The results demonstrate the significant expansion of the dataset, along with the number of images generated for each class and dataset type after applying various augmentation techniques. These details are provided in Table 4, which outlines the augmentation values obtained after the completion of the augmentation process.

Table 4. Augmentation values after augmentation

4.1 Results for RGB

The augmentation process is carried out based on the nature of the damage, and the images are transformed accordingly to match the specific augmentations applied. This procedure helps to enrich the existing dataset by generating a wider range of variations from the original data, making the model more robust and capable of handling different scenarios. Through these augmentations, the dataset becomes more comprehensive, allowing the model to better understand and detect the specific types of damage it is being trained on. The results of the augmentation process for the RGB dataset are illustrated in Figure 3, highlighting the enhanced diversity and variation achieved through augmentation techniques.

3.png

Figure 3. Results augmentation for dataset RGB

The training process for the model was conducted using Google Colaboratory. The results obtained can be seen in the images below.

Figure 4 shows the training log of a deep learning model over multiple epochs, with a focus on accuracy and loss for both training and validation datasets. The model was initially set to train for 200 epochs but was stopped early at epoch 37 due to the implementation of early stopping, which helps prevent overfitting by halting training when no further significant improvements are observed. By epoch 37, the model achieved a training accuracy of 98.62% and a validation accuracy of 96.88%, with a training loss of 0.0289 and a validation loss of 0.2019.

The two graphs illustrate the training and validation (test) accuracy and loss of a deep learning model over 37 epochs. In the accuracy graph, both training and test accuracies show an overall upward trend, with training accuracy consistently hovering around 95–100%, indicating effective learning from the training data. Test accuracy also remains high, despite some fluctuations, suggesting that the model generalizes well to unseen data. The training and validation accuracy over epochs are presented in Figure 5, demonstrating the model's strong and consistent performance.

In the loss graph, training loss remains low and stable throughout the epochs, indicating that the model is learning effectively without overfitting. The training and validation loss over epochs are shown in Figure 6, highlighting the model's stability and its ability to maintain low error rates across training iterations.

4.png

Figure 4. Training and validation process for RGB

5.png

Figure 5. Accuracy vs. epoch graph for RGB

6.png

Figure 6. Loss vs. epoch graph for RGB

The classification model was designed to distinguish between three categories of damage severity: Class 0 (No Damage), Class 1 (Minor Damage), and Class 2 (Severe Damage). These labels were used to interpret model predictions and assess performance metrics. The confusion matrix for the RGB dataset, which visualizes the distribution of correct and incorrect predictions across these classes, is presented in Figure 7.

Furthermore, the model’s balance between precision and recall for each class is depicted in Figure 8, through the recall versus precision curve for the RGB dataset. This curve provides additional insight into the model's classification performance, particularly its ability to maintain high precision and recall simultaneously.

7.png

Figure 7. Confusion matrix for RGB

8.png

Figure 8. Recall vs. precision curve for RGB

The confusion matrix and classification report show that the model has strong performance, with an overall test accuracy of 93.33%. The model perfectly classified instances of class 0 and class 2, but misclassified 2 instances of class 1 as class 2. Precision, recall, and F1-score for class 0 are all 1.00, while class 1 has a precision of 1.00 but a lower recall of 0.80, resulting in an F1-score of 0.89. Class 2 has a precision of 0.83 and recall of 1.00, with an F1-score of 0.91. Overall, macro and weighted averages for precision, recall, and F1-score are approximately 0.93, indicating consistent and reliable classification performance. The overall test accuracy achieved by the model for the RGB dataset is illustrated in Figure 9, providing a visual confirmation of the high accuracy obtained.

9.png

Figure 9. Test accuracy for RGB

4.2 Results for grayscale

The image augmentation process was applied to a dataset focused on detecting damage levels on surfaces. This augmentation process aims to generate new variations of the dataset by applying transformations. These variations enable the model to better recognize and detect different types of damage under various conditions. The results of the augmentation process for the grayscale dataset are illustrated in Figure 10, showing examples of the generated variations that help improve the model's robustness.

10.png

Figure 10. Results augmentation for dataset grayscale

11.png

Figure 11. Training and validation process for grayscale

Figure 11 shows the training log of a deep learning model over multiple epochs, focusing on grayscale image processing. The training process was initially set to run for 200 epochs but was halted early at epoch 68 due to early stopping criteria, indicating that further training would not yield significant improvement. By the end of the training, the model achieved a training accuracy of 99.03% with a corresponding loss of 0.027, while the validation accuracy reached 95.83% with a validation loss of 0.1311.

In this process, the results are better with RGB, even though some outcomes are similar. However, during training and validation, RGB showed superior performance compared to other methods.

The graphs illustrate the training and validation (test) accuracy and loss of a deep learning model using grayscale images over 68 epochs. In the accuracy graph, both training and test accuracies show a significant improvement, with training accuracy quickly reaching nearly 100% and test accuracy stabilizing above 95% after the initial epochs. This indicates effective learning and good generalization. The training and validation accuracy trends over the epochs are shown in Figure 12, providing a clear depiction of the model's progressive learning behavior.

12.png

Figure 12. Accuracy vs. epoch graph for grayscale

13.png

Figure 13. Loss vs. epoch graph for grayscale

The loss graph shows that training loss decreases steadily, remaining low and stable, suggesting that the model is efficiently minimizing error on the training data. The trend of training and validation loss over epochs is illustrated in Figure 13, highlighting the model’s ability to maintain low error rates throughout the learning process.

The confusion matrix and classification report indicate the performance of the model on the grayscale image test set. The confusion matrix, shown in Figure 14, reveals that the model correctly classified all instances of class 0 (10/10) and class 2 (10/10), but misclassified 2 instances of class 1 as class 2, achieving 8 correct classifications for class 1.

The classification report complements these findings, with perfect precision, recall, and F1-score (1.00) for class 0, while class 1 shows high precision (1.00) but a lower recall (0.80), leading to an F1-score of 0.89. Class 2 has a precision of 0.83, perfect recall (1.00), and an F1-score of 0.91.

14.png

Figure 14. Confusion matrix for grayscale

15.png

Figure 15. Recall vs. precision curve for grayscale

Further insight into the model's balance between precision and recall for each class can be seen in Figure 15, which displays the recall versus precision curve for the grayscale dataset.

The overall test accuracy for grayscale images, confirming the model’s strong classification performance, is illustrated in Figure 16.

16.png

Figure 16. Test accuracy for grayscale

4.3 Results for combine

The results from the augmentation process applied to this combined dataset resulted in a twofold increase in data size, bringing the total to 9,120 images. This significant expansion of the dataset enhances the model’s ability to generalize and improve performance across various conditions and scenarios. By creating diverse variations of the original images, the augmentation process strengthens the model’s capacity to recognize patterns, detect damage, and handle different levels of severity more effectively. The analyzed outcomes from this augmentation process are illustrated in Figure 17, highlighting the enriched dataset and its potential impact on model performance.

17.png

Figure 17. Results augmentation for dataset combine

This process shows the results of combining RGB and grayscale data, highlighting the differences that arise from this integration. Figure 18 shows the training log of a deep learning model over multiple epochs, showing the performance of the model using combined RGB and grayscale data. The training process was initially set for 200 epochs, but early stopping was triggered at epoch 58, indicating optimal performance without further improvement. By the end of training, the model achieved a training accuracy of 98.44% with a training loss of 0.0567, while the validation accuracy reached 97.22% with a validation loss of 0.0555. These results demonstrate the model's strong learning capability and effective generalization to validation data.

18.png

Figure 18. Training and validation process for combine

19.png

Figure 19. Accuracy vs. epoch graph for RGB combine

The graphs depict the training and validation (test) accuracy and loss of a deep learning model over 58 epochs, utilizing combined RGB and grayscale image data. In the accuracy graph, both training and validation accuracy steadily improve, with training accuracy approaching 100% and validation accuracy stabilizing around 95-97%. This demonstrates the model's ability to learn effectively and generalize well. The training and validation accuracy trends over epochs for the combined dataset are illustrated in Figure 19.

The loss graph shows a steady decline in training loss, indicating effective optimization and reduced errors in predictions. However, validation loss exhibits some fluctuation, with a general downward trend, suggesting occasional variability in the test data but overall good performance. The training and validation loss patterns are presented in Figure 20, offering additional insight into the model’s convergence behavior.

20.png

Figure 20. Loss vs. epoch graph for combine

21.png

Figure 21. Confusion matrix for combine

The confusion matrix for the combined dataset (Figure 21) illustrates the classification performance across three classes: Class 0 (No Damage), Class 1 (Minor Damage), and Class 2 (Severe Damage).

The model perfectly classified all instances of Class 0 and Class 2, with 20 correct predictions each. However, for Class 1, out of 20 total instances, 15 were correctly classified, 1 instance was misclassified as Class 0, and 4 instances were misclassified as Class 2. This indicates that the model had some difficulty distinguishing Class 1 from the others, though overall classification performance remains strong.

22.png

Figure 22. Recall vs. precision curve for combine

23.png

Figure 23. Test accuracy for combine

This resulted in class 1 having a recall of 0.75, indicating some difficulty in distinguishing class 1 from others. The overall test accuracy is 91.67%. The classification report shows precision, recall, and F1-score values across classes, with class 0 achieving near-perfect metrics and class 2 also showing high performance. Class 1, despite high precision, has a lower recall, affecting its F1-score. The macro and weighted averages for precision, recall, and F1-score are all around 0.92, reflecting robust overall model performance with consistent reliability across different classes. The balance between precision and recall across the classes for the combined dataset is visualized in Figure 22, through the recall versus precision curve. Meanwhile, the overall test accuracy for the combined dataset is depicted in Figure 23, confirming the high reliability of the model’s performance.

4.4 Comparison with baseline models

This section presents a comparative analysis between the proposed Inception ResNet V2 architecture and several baseline models to evaluate its effectiveness in hotspot detection on solar panels. A comparative analysis was conducted to evaluate the performance of Inception ResNet V2 against two baseline models, Inception V3 and Xception, using three types of datasets: RGB, grayscale, and combined. The comparative performance results across these models are summarized in Table 5, highlighting the testing accuracy and performance distinctions observed in the study.

Table 5. Comparison value in baseline models

The results revealed that Inception ResNet V2 achieved the highest performance, particularly on the combined dataset, with a testing accuracy of 91.66%, outperforming Inception V3 (66.66%) and Xception (85.00%). This model also demonstrated consistent results across training, validation, and testing phases, effectively avoiding overfitting, which was observed in other models, especially those trained with grayscale images. Although the model achieved high classification performance, it is important to acknowledge that the dataset consisted of only 570 images per class. This relatively limited data size may affect the generalizability of the model, especially when applied to more diverse or real-world datasets.

Future studies should consider expanding the dataset and incorporating additional data augmentation techniques to enhance the model’s robustness and reduce potential overfitting. The use of the combined dataset proved to enhance feature richness, and only the Inception ResNet V2 architecture was able to process it effectively without compromising generalization capabilities. This is further supported by the "Bestfit" remark assigned exclusively to this model, whereas the others showed signs of overfitting. Therefore, Inception ResNet V2 is considered the most reliable and stable architecture for hotspot detection on solar panels based on the observed performance outcomes.

Table 6. Prediction results for RGB dataset

Classification	Inception ResNet V2
No Damage (All predictions were analyzed and found to be accurate)	1-1.png 1-2.png
Minor Damage (The analysis showed that all predictions were accurate)	2-1.png 2-2.png
Severe Damage (Every prediction was reviewed and proven to be correct)	3-1.png 3-2.png

Although the dataset was identical across experiments, performance differences were mainly due to architectural variations, as each model processes features differently. This highlights the crucial role of model design in classification performance. The prediction results for the RGB dataset across models are summarized in Table 6.

Building upon these findings, the next step involves conducting further tests to predict and validate the real-world performance of the proposed architecture. These evaluations are essential for determining the model's ability to generalize and accurately detect the targeted patterns. The results from such tests will provide valuable insights into the overall reliability, robustness, and precision of the model, guiding potential improvements for future implementations. The prediction results specifically for the grayscale dataset are presented in Table 7, offering a detailed assessment of the model's performance on grayscale images.

Table 7. Prediction results for grayscale dataset

Classification	Inception ResNet V2
No Damage (All predictions were analyzed and found to be accurate)	711.png 712.png
Minor Damage (The analysis showed that all predictions were accurate)	721.png 722.png
Severe Damage (Every prediction was reviewed and proven to be correct)	731.png 732.png

Despite expectations that the combined dataset would enhance model performance by incorporating both RGB and grayscale representations, the results show that this dataset did not consistently outperform the individual formats across all architectures. This may be attributed to potential feature conflicts or redundancies, where overlapping or contradictory information from RGB (color-based features) and grayscale (intensity-based features) can lead to ineffective learning, particularly in models not optimized for multi-channel input integration. For instance, Inception V3 and Xception exhibited lower testing accuracy with the combined dataset compared to their performance with RGB-only or grayscale-only inputs. This finding suggests that the success of combining multiple image formats depends heavily on the model’s ability to process heterogeneous feature types effectively. The prediction results for the combined dataset across the evaluated architectures are presented in Table 8, summarizing the classification outcomes.

Table 8. Prediction results for combine dataset

Classification	Inception ResNet V2
No Damage (All predictions were analyzed and found to be accurate)	811.png 812.png
Minor Damage (The analysis showed that all predictions were accurate)	821.png 822.png
Severe Damage (Out of 8 test data sample, 1 was misclassified)	833.png 834.png

Levene’s Test for Equality of Variances indicated a significant difference in variance (F=13.303, p=0.000), suggesting that the assumption of equal variances is not met. Therefore, the t-test results under “equal variances not assumed” were used. The test showed a statistically significant difference between the two groups with t=2.587, df≈37.585, and p=0.014 (p<0.05). The mean difference in accuracy was 5.50486%, with a 95% confidence interval ranging from 1.19626% to 9.81347%.

These findings clearly demonstrate that the RGB dataset significantly outperforms the grayscale dataset in hotspot detection accuracy. The lower variance and higher mean accuracy of the RGB group suggest that using RGB imagery results in more stable, reliable, and consistent model performance for solar panel anomaly classification compared to grayscale input.