Deep Learning Based Remote Sensing Image Annotation Based on Semantic and Contextual Information

Image annotation is a crucial process that underpins numerous computer vision applications, from medical imaging and remote sensing to smart agriculture and mobile visual search. The reviewed studies leverage deep learning and hybrid methods to automate or accelerate this process, aiming to reduce manual effort and improve annotation precision. Despite these advancements, several limitations and challenges persist that necessitate further exploration.

Fernandes et al. [11] presented an innovative annotation approach for video capsule endoscopy (VCE) images, which is particularly beneficial while manual labeling is time-consuming and prone to error. They employed a Siamese network with ResNet-18 for image similarity matching, achieving an accuracy of over 97%. Their method streamlines medical annotation by reducing reliance on manual tools and enabling efficient CBIR in clinical workflows. Gautam and Khanna [12] addressed the limitations of text-based image retrieval by utilizing VGG16 and ResNet-50 to extract deep features for CBIR. Their system enhances precision, narrows the semantic gap, and provides an intuitive interface suitable for applications such as medical retrieval and visual search on mobile devices. Palekar [13] introduced a hybrid deep learning model that combines CNNs with biologically inspired feature selection for automatic image annotation. Using techniques such as LBP, Slantlet transform, and a red colobus monkey-inspired algorithm, the system achieved up to 99.2% accuracy on standard datasets, providing a creative and effective annotation pipeline. Meenakshi et al. [14] focused on lightweight CBIR for mobile devices using AdaBoost. By refining weak classifiers and emphasizing misclassified images, their model achieved a precision of 95.6% while remaining computationally efficient, making it ideal for real-time mobile image search. Zhang et al. [15] developed an interactive medical image annotation framework using Attention U-Net and geodesic distance guidance. Their two-stage segmentation process reduces manual effort while maintaining high accuracy on prostate MRI data, supporting faster and more efficient dataset labeling.

Liu et al. [16] proposed AIO², a system that corrects noisy labels in remote sensing segmentation using adaptive feedback and a mean-teacher model. Applied to building datasets, it improved annotation quality without requiring manual re-labeling, making it ideal for large-scale Earth observation. Song et al. [17] introduced a robust training method for noisy medical labels called Confidence Regularized Co-Teaching. The model filters uncertain labels by focusing on confident disagreements, yielding strong results even with imperfect annotations, which is key for scalable clinical AI. Mamat et al. [18] developed a YOLO-based annotation tool for agricultural images, which classifies oil palm ripeness with an accuracy of over 98%. The system supports smart farming by enabling fast and reliable fruit classification using transfer learning on RGB images. He et al. [19] developed PhenoLearn, a toolkit designed for biologists to annotate images of natural specimens. With a GUI and a deep learning backend, it enables non-experts to accurately label and segment traits, bridging the gap between biology and AI tools, as tested on bird plumage. Beck et al. [20] investigated the combination of ChatGPT-4V with human annotators for labeling satellite images. Their hybrid system reduced costs by 50% with minimal loss of accuracy, demonstrating how large language models can support scalable and cost-effective annotation workflows.

Lotfi et al. [21] proposed a graph-based method for automatic image annotation, modeling relationships between tags using GNNs. Tested on Corel5k and ESP Game datasets, the method improved annotation precision by capturing tag dependencies, enhancing multi-label performance. Wang et al. [22] developed AIDE, a framework that achieves near-supervised segmentation quality using only 10% labeled data. Applied to breast tumor datasets, AIDE reduces reliance on costly manual annotations while maintaining high diagnostic accuracy. Neptune and Mothe [23] used scientific literature to enrich satellite deforestation image annotations. By combining deep change detection with semantic embedding, their system generates context-aware labels, enhancing the interpretability of environmental monitoring tools.

Huang et al. [24] developed a deep learning framework for multi-label remote sensing annotation that utilizes attention and feature fusion. The model performs well on complex scenes by capturing label correlations and detecting small objects, boosting annotation accuracy in RS imagery. Hua et al. [25] introduced FESTA, a weakly supervised segmentation method that works with sparse scribble annotations. It combines spatial and feature regularization to achieve strong performance with minimal input, making it a cost-effective solution for labeling aerial images. To address the noise, Zhang et al. [26] introduced Deep Multi-Similarity Hashing (DMSH) for RSI retrieval by fusing hashing and spatial information. For the UCM dataset, it yielded a mean average accuracy of 0.97. For cross-modal retrieval, Cheng and Zhang [27] presented a Deep Semantic Alignment Network (DSAN), which was evaluated on many captioned datasets, such as NWPU-RESISC45-Captions and UCMerced-LandUse-Captions. Through semantic alignment, it improves cross-modal retrieval and attains above 90% classification accuracy on the UCMerced-LandUse dataset. Its efficacy may be limited by its poorer accuracy in visually comparable categories, despite its strong performance. For RSI retrieval, Maurya et al. [28] proposed an adaptive DL-based model to address the drawbacks of low retrieval, rigidity, and inefficiency. They examined how well the various transfer learning models performed in image retrieval on the UCM dataset. For the 21-class classification, the accuracy of 95.07% for VGG19 was higher than 93% for VGG16 and 91% for ResNet.

Various YOLO based schemes has been presented for object detection and annotation in RSIs. Blushtein-Livnon et al. [29] examined how annotation tactics, dataset imbalances, and annotators' experience impact the accuracy of remote sensing labelling in their lab experiment. Professionals and non-experts labeled aerial images in ArcGIS Pro to identify and segment small PV panels. It is found that people are better at detection than segmentation. Annotators were more likely to make Type II mistakes (missing items) than Type-I errors (false detections) under different situations, suggesting a cautious approach that favored underestimating. Extreme imbalance negatively affected accuracy, whereas datasets with balanced target-to-background ratios performed more effectively. Interestingly, experience had no influence and occasionally led to overconfidence in segmentation. These results suggest that remote sensing requires improved annotation methodologies to meet the growing demand for high-quality training data.

Wang et al. [30] introduced AG-YOLO, a remote sensing-optimized attention-guided object identification system. Adding a rotation parameter to YOLOv10 and expanding the dual label assignment mechanism improves oriented object recognition. A separate attention branch suppresses complicated background noise and concentrates on key foreground details. Additionally, a three-stage curriculum learning technique enables the model to learn from easier examples before processing more complex data. Zhao et al. [31] introduced SCENE-YOLO. This novel detection framework integrates scene supervision into the YOLOv8 architecture, resulting in reduced processing latency and improved detection accuracy compared to other state-of-the-art approaches on the DOTA dataset. Their technique enhances detection by modeling object instance connections and utilizing background information that other approaches overlook. A transformer-based scene information network adds semantic context to feature extraction, and an omnidimensional dynamic convolution method modifies feature weighting. In complex scenarios, a prototype-based scene label generation method and a scene-assisted detection head improve classification accuracy. DOTA and DIOR experiments showed the model outperformed standard detection networks.

Hu et al. [32] employed CM-YOLO, a cloud- and mist-affected remote sensing image model, to enhance detection in challenging weather conditions. A component-decoupling background suppression module improves target-background contrast, while a local-global semantic mining module combines convolutional networks and selective attention to extract rich contextual data. CM-YOLO exceeded multiple detectors in terms of accuracy, recall, and mAP. Qian et al. [33] proposed IPS-YOLO, a pseudo-fully supervised training system, to address issues in weakly supervised object detection (WSOD). Their technique solves partial object identification and static pseudo ground truth concerns in prior methods. They employed category confidence-guided filtering and weighted synthesis to create a more comprehensive PGT and utilized iterative refinement during training. Proposal creation, increased sample selection, and pseudo-label definition further boosted performance. Comparative investigations demonstrated that IPS-YOLO outperformed WSOD models on numerous benchmark datasets.

Transfer learning for remote sensing object identification was examined by Pandilova et al. [34] using YOLOv8. They compared models trained from scratch with COCO-trained and DIOR- and Ships-tuned models. Transfer learning enhanced detection accuracy, precision, and recall, especially for tiny and similar items. Confusion matrix analysis revealed higher category discrimination, indicating that fine-tuning pre-trained models is beneficial for remote sensing. Finally, Jin et al. [35] introduced MTGS-YOLO, a task-balanced object identification method for remote sensing photography with complicated backdrops and high-resolution data. Their concept utilizes a multi-transformer architecture to capture both global and local information, and employs a GELAN for enhanced flexibility and computational performance. Small item recognition is improved by removing irrelevant background characteristics using a Spatial Context-Aware Module (SCAM). In the DIOR and NWPU dataset experiments, MTGS-YOLO outperformed previous algorithms in terms of detection accuracy, robustness, and generalization.

From the survey of recent techniques of image annotations, the following research gaps are identified:

• While current methods improve annotation accuracy and efficiency, many rely heavily on pre-trained networks without sufficient domain adaptation, which limits their generalization across tasks.
• Few approaches address annotation in low-resource environments or explore how to reduce model complexity for edge deployment.
• Though some works focus on noisy labels, robust handling of label noise under limited supervision remains underexplored.
• Most solutions are dataset-specific, with inadequate validation on diverse or real-world datasets.
• There's also limited integration of multimodal cues (e.g., text, metadata) for enhanced annotation quality.
• Scalability and usability for non-experts, particularly in fields such as biology or agriculture, are often overlooked.
• Additionally, semi-supervised and active learning-based annotation pipelines are still evolving and lack standardization.
• Interactive systems exist, but they often fail to optimize user feedback integration or reduce cognitive load.
• Ethical considerations, such as annotation bias and privacy in medical datasets, are rarely discussed.
• Finally, real-time, explainable annotation frameworks with human-AI collaboration capabilities remain an open challenge.

4.1 System setup

The suggested system is implemented using Python programming on an NVIDIA GPU with 128 GB of RAM, 32 GB of graphics memory, and a Windows operating environment. The parameter specifications for YOLOv8 are listed in Table 1.

Table 1. Parameter specification of YOLOv8

4.2 Performance metrics

The effectiveness of the proposed system is evaluated based on recall, precision, F1-score, accuracy, and mean average precision (mAP) as given in Eqs. (4)-(8), respectively. Here, TP_V, TN_V, FP_V, FN_V, and AP depict the actual positive value, true negative value, false positive value, false negative value, and area under the recall-precision curve for N classes. The recall and precision offer the quantitative and qualitative measures of the retrieval results. The accuracy represents the overall accuracy; the F1-score offers a quality-quantity balance; and the mAP provides a rank-based balance between precision and recall.

Recall $=\frac{T P_V}{T P_V+F N_V}$     (4)

Precision $=\frac{T P_V}{T P_V+F P_V}$  (5)

Accuracy $=\frac{T P_V+T N_V}{T P_V+F N_V+F P_V+T N_V}$  (6)

$F 1-$ score $=\frac{2 * { Recall } * { Precision }}{{ Recall }+ { Precision }}$  (7)

$m A P=\frac{1}{N} \sum_{i=1}^N A P_i$ (8)

4.3 Dataset

To assess the performance of the proposed CBIR approach, experiments were conducted using the UC Merced Land Use Dataset [1]. This dataset comprises 21 distinct categories of land use, each containing 100 images with a resolution of 256 × 256 pixels. The model uses 70% of the data for training and 30% for testing purposes. The images were sourced from the USGS National Map's Urban Area Imagery, capturing various urban regions across the United States. A few example images from the dataset are shown in Figure 4.

Figure 4. UCM sample images

4.4 Discussions on results

The visualizations of the object annotation and detection are shown in Figures 5-7, respectively, for YOLOv8, YOLOv8+DCN-C2F, and YOLOv8+DCN_C2F+SCConv-C2F. The YOLOv8 detects the salient object but misses finer objects with occluded positions, as shown in Figure 5. The DCN and SCConv help detect fine and multiscale objects in images by providing spatial connectivity in the inter-channel positions of the objects, as illustrated in Figure 6 and Figure 7, respectively.

Figure 5. Object detection for sample image using YOLOv8

Figure 6. Object detection for sample image using YOLOv8+DCN_C2F

Figure 7. Object detection for sample image using YOLOv8+DCN_C2F+SCConv_C2F

Table 2. Comparative analysis of the proposed method

The effectiveness of image object annotation is evaluated across various YOLO architectures, including YOLOv3, YOLOv5, and YOLOv8, as shown in the Table 2. The traditional YOLOv3 achieves a recall rate of 77.50%, a precision of 85.80%, an F1-score of 81.41%, and an accuracy of 80.10% on the UCM dataset. The YOLOv5 provides improved accuracy compared to YOLOv3, resulting in a recall of 86.20%, a precision of 87.50%, an F1-score of 86.84%, and an accuracy of 87%. The YOLOv8 achieved a recall rate of 89.20%, a precision of 94.20%, an F1-score of 91.62%, and an accuracy of 91.80%, which demonstrates a notable improvement over YOLOv3 and YOLOv5, owing to its ability to learn the finer details of image objects. The DCN-C2F allows the model to adaptively adjust its receptive fields, enabling it to capture finer object details and better understand spatial context. The YOLOv8+DCN_C2F achieves a recall rate of 94.40%, precision of 94.30%, F1-score of 94.35%, and accuracy of 94.20%. The SCConv-C2F enhances the network’s ability to extract and utilize semantic relationships across different channels. The YOLOv8+SCConv_C2F achieves an overall recall of 97.50%, precision of 97.10%, F1-score of 97.30%, and accuracy of 97.10%. The integration of DCN and SCConv in YOLOv8 results in improved recall of 98.50%, precision of 98.40%, F1-score of 98.45% and accuracy of 98.15% which shows a crucial boost for remote sensing object detection and annotation over traditional models. The visualizations of the comparative results for the various models are shown in Figures 8-11.

Figure 8. Recall comparison of different methods

Figure 9. Precision comparison of different methods

Figure 10. F1-score comparison of different methods

Figure 11. Accuracy comparison of different methods

Table 3. Performance comparison of the proposed YOLOv8 for different learning algorithms

Table 3 presents the performance comparison of the proposed YOLOv8 framework under different learning algorithms for remote sensing image object detection and annotation. With the Adam optimizer, the baseline YOLOv8 achieved an accuracy of 91.80% with a mAP of 0.92, which was further enhanced to 94.20% (mAP 0.94) when deformable convolution layers (DCN_C2F) were integrated. The inclusion of spatial channel-wise convolution (SCConv_C2F) produced a significant boost, reaching 97.10% accuracy and 0.97 mAP, while the combined DCN_C2F + SCConv_C2F configuration delivered the best results at 98.15% accuracy and 0.99 mAP. In contrast, the SGDM optimizer yielded slightly reduced performance, with YOLOv8 scoring 88.80% accuracy and 0.89 mAP, improving to 95.00% accuracy and 0.96 mAP for the hybrid configuration. The RMSProp optimizer showed the lowest overall results, with the baseline YOLOv8 at 86.00% accuracy and 0.86 mAP, while its best performance under DCN_C2F + SCConv_C2F reached 92.50% accuracy and 0.93 mAP. These outcomes indicate that while architectural enhancements such as deformable and spatial convolutions substantially improve detection accuracy across all optimizers, Adam consistently outperforms SGDM and RMSProp by margins of 3–5% in accuracy and up to 0.03 in mAP, demonstrating its robustness for precise object detection and annotation in complex remote sensing imagery.

The effectiveness of the system is compared with the traditional state-of-the-art using the UCM dataset for object retrieval and annotation, as shown in Table 4, based on average accuracy and mAP. The proposed method shows a remarkable improvement over existing techniques in terms of both accuracy and mAP. Compared to DSAN, which achieved 90% accuracy, and Maurya et al.'s use of traditional CNN architectures, such as VGG19 (95.07%), the advanced integration of SCConv and DCN modules into the YOLOv8 architecture results in a significant boost. Specifically, the final configuration—YOLOv8 combined with DCN_C2F and SCConv_C2F achieves 98.15% accuracy and 0.99 mAP. This marks an 8.15% improvement in accuracy over DSAN and a 3.08% increase over the best-performing CNN baseline (VGG19)—moreover, the mAP increases by 2% over the YOLOv8 baseline and by 2.06% over DMSH. The novel YOLOv8 offered an improved mAP compared to existing YOLO-based frameworks, such as AG-YOLO, CM-YOLO, and YOLO-COCO, which provided mAPs of 0.80, 0.85, and 0.76 for remote sensing images. These enhancements demonstrate the effectiveness of combining deformable convolution and selective context convolution modules, enabling the network to capture spatial context and object deformations more effectively, resulting in more accurate detection and annotation.

Table 4. Comparison of proposed RSI retrieval and annotation with traditional techniques