Automated Detection of Construction Opening Safety Risks Using Computer Vision

The construction industry is one of the least digitized sectors globally and, due to its highly labor-intensive nature, is also a high-risk industry where worker injuries frequently occur. Common causes of injuries include improper working postures, manual handling of heavy objects, and high-intensity operations [1]. Among these, Falls from Heights (FFH) are considered one of the primary causes of severe injuries and fatal accidents on construction sites. Studies showed that falls from heights account for 48% of all construction-related accidents, with scaffolding and roofing operations posing particularly high risks [2].

A study focusing on construction sites in Hong Kong analyzed the root causes of fall-from-height incidents and found that a lack of adequate protective equipment at open edges often led to accidents during high-elevation work. The study suggested that improving site design and providing better safety measures could effectively reduce the risk of falls [3]. Additionally, research highlighted that falls from heights represented a significant portion of construction accidents, underscoring the importance of standardized fall protection systems, especially in edge work operations [4].

The impact of fall-from-height accidents not only results in immediate injuries but also brings about substantial social and economic losses, particularly in cases where workers fall from unprotected edges or heights. In response, some studies proposed enhancing workers' safety awareness and strictly enforcing the correct use of protective equipment. These measures were considered crucial for effectively reducing the occurrence of such accidents [5].

With the rapid development of Artificial Intelligence (AI), advanced technologies such as Computer Vision (CV) and Natural Language Processing (NLP) were progressively applied to the field of construction management, aiming to assist safety personnel in monitoring construction site safety [6-9]. The high accident rate in the construction industry caused significant social harm, making the effective implementation of safety management a key focus. For instance, Fang et al. [10] developed an Image-Text Semantic Matching (ITSM) model to evaluate the semantic similarity between visual and textual features, determining whether workers' behaviors in images violated the safety regulations recorded in text. However, the model's binary output format limited scalability. Ding et al. [11] proposed a Visual Question Answering (VQA)-based inference framework for training and application in safety compliance checks. Emerging information technologies such as the Internet of Things (IoT) and computer vision were also applied to on-site safety monitoring [12]. These technologies require recording and storing large amounts of data, such as digital images and incident reports, as a basis for preventing future accidents [13]. However, extracting insights from massive datasets for safety management and decision-making remained a challenge [14].

Research showed that enhancing the standardization and transparency of reports helped strengthen communication effectiveness and consistency within and outside organizations. Standardized reports provided a reference framework, simplified reporting processes, and ensured the accuracy and completeness of information transmission [15]. Furthermore, applying digital and intelligent technologies to optimize reporting processes through data management further enhances decision-making efficiency and information transparency. This was particularly important in construction management, where digital reporting improved information consistency and simplified the overall process [3]. Specifically, customized reporting processes allowed digital architectures to adjust according to the needs of multiple stakeholders, ensuring that reports accurately met various demands, especially in engineering projects involving multi-party collaboration [16]. Overall, enhancing standardization and transparency made reporting processes more consistent across different business scenarios, enabling more efficient and consistent satisfaction of diverse management requirements in construction project management [15].

Although recent studies have increasingly applied artificial intelligence to construction safety management and have developed image recognition–based risk detection systems, two major limitations remain. First, most existing image-text semantic matching (ITSM) approaches focus only on single-level semantic alignment, often overlooking subtle distinctions related to the correctness of installation and regulatory compliance of fall protection facilities around openings. This results in limited precision and insufficient semantic depth in the detection outcomes. Second, current systems emphasize violation detection but lack mechanisms for tracking subsequent corrective actions, making it challenging to implement closed-loop risk management and verification.

To address these issues, this study proposes an automated detection system that integrates YOLO for object detection and CLIP (Contrastive Language Image Pretraining) for image-text semantic matching. The system improves semantic resolution and detection accuracy while incorporating a report generation and feedback module to support real-time monitoring and continuous risk tracking. This integrated approach addresses the shortcomings of existing AI-based safety systems by enabling both precise risk identification and closed-loop safety management.

3.1 Object Recognition model

This section introduces an Object Recognition model for image recognition of construction site openings and fall protection facilities. The objective is to enhance the model's generalization ability through a diverse training dataset. This dataset includes images of construction site openings and fall protection facilities collected from open computer vision platforms. Images are annotated using a randomized method (detailed in Section 3.1.1) and are divided into training, testing, and validation sets. This partition helps fine-tune parameters during training to prevent overfitting while ensuring robust performance on unseen data.

1.png

Figure 1. Model training process

To improve recognition performance, the training data undergoes preprocessing and data augmentation. The training process is conducted using the PyTorch framework. Various evaluation metrics are introduced to evaluate the model's performance comprehensively. The framework is illustrated in Figure 1.

3.1.1 Hazard image dataset

The dataset employed in this study primarily consists of images of construction site openings captured on-site, along with images of fall protection facilities—including guardrails, opening covers, and safety nets—collected from open-source computer vision platforms. All images were annotated using labelImg, and the dataset was randomly divided into training, validation, and testing subsets in a ratio of 8:1:1. The training set was used to facilitate model learning, the validation set was utilized during the training phase to monitor performance, adjust hyperparameters, and prevent overfitting, while the testing set was reserved for final evaluation to assess the model’s generalization on unseen data. The original dataset included 184 on-site images of construction openings and 2,163 of fall protection facilities. After applying data augmentation techniques, the total number of images in the dataset increased to 7,041.

To enhance the model's generalization capability, a series of preprocessing and data augmentation operations were applied to the dataset. First, all training images were subjected to auto-orientation to ensure consistency in image direction. Second, to meet the input requirements of the neural network, the dimensions of each image were uniformly resized to 640x640 pixels. In terms of data augmentation, to simulate variations in perspectives and scales, the following methods were applied to each training sample, generating three transformed images:

Horizontal and Vertical Flipping: Simulates changes in object orientation within the scene.

90-Degree Rotation: Includes clockwise, counterclockwise rotations, and vertical flipping to enhance the model's robustness to directional changes.

Random Cropping: Randomly crops images within a scaling range of 0% to 20%, helping the model learn to recognize objects from partial perspectives.

3.1.2 Model development and training

This study constructed an object detection model based on YOLOv8 using the PyTorch framework in a Python 3.11 environment within PyCharm. All experiments were conducted on a PC equipped with the following specifications: CPU: 12th Gen Intel(R) Core(TM) i7-12700KF 3.60 GHz, GPU: NVIDIA GeForce RTX 3070-12G, and RAM: 32 GB.

Table 3. Key hyperparameters

Before training the YOLO model, a pre-trained base model was selected to enhance efficiency and performance. The network architecture was adjusted as needed, and key hyperparameters like learning rate and batch size were optimized (Table 3). During training, the model processed image data to generate predictions, calculated loss values against ground truth, and updated parameters via backpropagation. This iterative process continued until the performance was satisfactory or the set number of epochs was reached.

The table presents a series of key parameters configured during the training of the YOLOv8 and YOLOv10 models. These parameters are carefully selected to optimize the training process and enhance model performance, as explained below:

Model:

This study will test all models from YOLOv8 and YOLOv10.

Batch Size:

Set to 8, representing the number of images processed in each iteration. A large batch size may result in more stable gradient estimation, but can cause memory overflow on limited hardware. Conversely, a small batch size may lead to high gradient variance, affecting the stability of model convergence. Empirical testing revealed that a medium-sized batch provides a good trade-off between training efficiency and resource constraints, ensuring consistent training progress.

Epochs:

Set to 100, indicating that the training data is used in full 100 times. Insufficient epochs may result in underfitting, where the model fails to learn adequate features. However, excessively high epochs may lead to overfitting, particularly without regularization or early stopping mechanisms. Appropriate epoch selection should be guided by the model’s validation performance trend and dynamically adjusted based on convergence behavior.

Optimizer:

The AdamW optimizer was employed, which uses adaptive learning rates and is designed to improve model performance by adjusting parameters dynamically during training. Compared to traditional optimizers such as SGD or standard Adam, AdamW improves convergence efficiency in high-dimensional parameter spaces and helps reduce overfitting, which is especially important when training deep neural networks on safety-critical datasets.

Loss Function:

Cross-entropy loss was used, a function commonly applied to classification tasks to measure the difference between the predicted probability distribution and the actual labels.

Intersection over Union (IoU):

Set to 0.7, this metric evaluates the overlap between predicted and ground truth bounding boxes. If set too low, the model may overestimate accuracy, resulting in false positives. If set too high, it may miss partially correct predictions, increasing false negatives.

Learning Rate:

Set to 0.00125, a key factor determining the magnitude of weight updates. A well-chosen learning rate helps the model learn effectively, avoiding oscillations or excessively slow convergence during training. A high learning rate may cause unstable gradients or oscillation, especially in the early training stages. On the other hand, a very low learning rate slows down convergence, preventing the model from reaching optimal performance within a reasonable timeframe. Selecting a moderate and dynamically adjustable learning rate contributed to training stability and convergence efficiency in this study.

3.1.3 Evaluation metrics

In object detection models, commonly used evaluation metrics include:

Intersection-over-Union (IoU):

To evaluate the model's performance, ground truth bounding boxes (manually annotated) are compared with the predicted bounding boxes. The Intersection-over-Union (IoU) metric is calculated as the ratio of the overlapping area between the predicted bounding box and the ground truth bounding box (intersection, ∩) to the combined area of both bounding boxes (union, ∪), as shown in Eq. (1):

$I o U=\frac{\text { Area of Overlap }}{\text { Area of Union }}=\frac{\left|B_{\text {pred }} \cap B_{g t}\right|}{\left|B_{\text {pred }} \cup B_{g t}\right|}$     (1)

where,

$B_{\text {pred }}$ denotes the predicted bounding box generated by the object detection model.

$B_{g t}$ denotes the ground truth bounding box annotated in the dataset.

Average precision (AP):

Precision and recall are calculated for various confidence score thresholds, allowing the precision-recall curve to be plotted. AP is computed as the area under the precision-recall curve, as defined in Eq. (2):

$A P=\int_0^1 P(r) d r$     (2)

where,

P(r) denotes the precision as a function of recall r, describing the model’s precision at a given level of recall.

r represents the recall, defined as the ratio of correctly predicted positive instances to all actual positive instances.

Mean average precision(mAP):

To evaluate the model's overall performance, mAP is calculated as the mean of the AP values across all categories. In object detection, mAP is traditionally computed for a single IoU threshold (denoted as mAP_IoU). In YOLO series models $s m A P_{50}$ and $m A P_{50-95}$, are commonly used to represent precision metrics, as shown in Eq. (3):

$m A P=\frac{1}{C} \sum_{C=1}^C A P_C$     (3)

where,

C is the total number of categories.

$A P_c$ is the average precision for category c.

Accuracy:

Accuracy is the ratio of correct predictions to the total number of predictions, as shown in Eq. (4):

Accuracy $=\frac{T P+T N}{T P+T N+F P+F N}$     (4)

Precision:

Precision is the ratio of true positive predictions to the total number of positive predictions, as shown in Eq. (5):

Precision $=\frac{T P}{T P+F P}$     (5)

Recall:

Recall is the ratio of true positive predictions to the total number of actual positive cases, as shown in Eq. (6):

Recall $=\frac{T P}{T P+F N}$     (6)

F1-score:

The F1-score combines precision and recall into a single metric, calculated as their harmonic mean, as shown in Eq. (7).

$F 1-$ Score $=2 * \frac{\text {Precision} * \text {Recall}}{\text {Precision}+ \text {Recall}}$     (7)

Notes:

True Positive (TP): The number of positive samples correctly predicted as positive.

False Positive (FP): The number of negative samples incorrectly predicted as positive.

True Negative (TN): The number of negative samples correctly predicted as negative.

False Negative (FN): The number of positive samples incorrectly predicted as negative.

3.2 Image-text semantic matching

This section explains the prepared textual descriptions, including installation requirements and safety regulations for fall protection facilities, which serve as the basis for image comparison. It also details the threshold settings to ensure the model accurately filters out non-compliant images. Additionally, the image-to-semantic matching process is described as enhancing detection accuracy.

3.2.1 Text preparation

This study aims to address safety management issues on construction sites, particularly focusing on identifying openings and fall protection facilities. To achieve this, content related to fall hazard prevention facilities was extracted from the Standards for Construction Safety and Health Facilities. Since the original legal texts are often lengthy, they were transformed into a format more suitable for CLIP model analysis. This process involved simplifying the detailed descriptions in the regulations into shorter sentences, enabling the model to perform ITSM more effectively.

Research indicated that the CLIP model performed more efficiently in learning and predicting image content when processing concise and precise textual descriptions [42].

Data collected from social media platforms further demonstrated how short texts improved image-text alignment efficiency in various contexts, especially descriptive text and image pairing [43]. A novel contrastive training method introduced in recent studies optimized representation alignment through large-scale image-caption and text-text pairs, allowing the model to excel in text-image and text-text matching tasks. This highlighted the importance of concise sentences in enhancing the performance of the CLIP model [44]. The training and application of the CLIP model consistently demonstrated its superior ITSM capabilities, particularly in zero-shot learning scenarios when dealing with new images similar to the training data distribution [42]. Noted that the CLIP model, through its contrastive learning framework, effectively aligns image and text representations, enabling accurate image recognition and classification.

3.2.2 Threshold setting and performance analysis

To achieve ITSM for assisting construction site safety inspections, this study developed a text description database based on relevant safety regulations. This database serves as the semantic basis for the CLIP model, containing descriptions of various non-compliance scenarios related to construction opening protection measures. These descriptions cover common safety violations, such as "excessive guardrail spacing" and "incorrect guardrail installation". During system operation, the CLIP model processes input images and selects the most appropriate violation description from the database to determine whether a given image corresponds to a specific non-compliance scenario.

To ensure the accuracy and reliability of the ITSM process, this study conducted threshold optimization and sensitivity analysis for the CLIP model. The threshold value directly influences the sensitivity and accuracy of the violation detection system:

Excessively high threshold: If the threshold is set too high, the CLIP model will only classify an image as a violation if its similarity score exceeds the defined threshold. This could result in false negatives, where certain non-compliant scenarios remain undetected due to lower similarity scores. Consequently, some potential hazards may be overlooked by on-site personnel, compromising the comprehensiveness of safety management.

Excessively low threshold: If the threshold is set too low, even images with relatively low similarity scores may be classified as violations, leading to false positives. This would generate an excessive number of reports containing non-violative conditions, thereby increasing the workload of safety inspectors and reducing the practical utility of the system.

The image-text matching process is illustrated in Figure 2. The system first captures images of the construction site, which are processed through an image encoder to extract visual feature vectors. Simultaneously, predefined textual descriptions of potential safety violations (e.g., "the spacing between guardrails is too wide") are input into a text encoder to generate corresponding textual feature vectors.

After feature extraction, visual and textual features are projected into a shared embedding space. Within this space, the system performs semantic matching by calculating the cosine similarity between the feature vectors. The similarity score represents the semantic alignment between the site image and the violation descriptions. In the example shown in Figure 2, the calculated cosine similarity is 0.72. If this score exceeds the system's predefined threshold, the system automatically classifies the image as a "violation."

2.png

Figure 2. Semantic matching process for construction site safety compliance using CLIP

3.3 System framework and workflow

Figure 3 illustrates the complete system architecture for the automated construction safety inspection process, which is divided into four main stages: Data Input, Object Detection, Semantic Matching, and Risk Report Generation, achieving real-time monitoring and risk management.

First, the system starts with the Input Phase, where users upload construction site images as the foundational data for subsequent analysis. These images undergo Standardize Image processing to ensure consistent size and resolution, enhancing the accuracy and stability of the model analysis. After standardization, the images enter the Detect Image process, where the YOLO model automatically detects construction openings, guardrails, fences, and fall protection installations, marking potential hazard areas for further evaluation.

3.png

Figure 3. System framework

Next, as shown in Figure 3, the detected images proceed to the Vectorize stage, where image features are encoded into high-dimensional vectors and compared with predefined safety regulations. This process utilizes Similarity Comparison to calculate the cosine similarity between image and text descriptions, determining whether the site installations comply with safety standards. If the similarity score exceeds the system-defined Threshold Standard, the image is flagged as a "Potential Violation" for further processing.

Following the similarity matching, the system automatically enters the Generate Risk Report stage, generating structured reports for the identified potential hazards. These reports detail the location, type, and risk level of the non-compliant installations. The generated reports are then presented in the Review Report interface, allowing site management personnel to review and handle the detected issues, enhancing the transparency and traceability of inspection results.

Finally, after the inspection and report generation processes are complete, the system transitions to the End phase, indicating the completion of that batch of inspections. As depicted in Figure 3, this entire workflow achieves full automation from data collection to semantic matching, risk identification, report generation, and review. It not only reduces the costs associated with manual inspections but also enables real-time detection of potential construction hazards, ensuring site safety and improving management efficiency.