Application of Deep Learning-Based Multi-Scale Feature Fusion in the Visual System of Precision Welding Robots

2.1 Data acquisition

Experimental sample acquisition requires the construction of a dynamic dataset covering multiple working conditions and defect types. This paper adopts a visual inspection unit composed of a binocular vision sensor and a high-speed industrial camera, integrated into the end-effector of the welding robot. During the welding process, RGB images and depth information of the weld area are synchronously collected at a frame rate of over 200 fps, covering welding processes such as arc welding and laser welding, base materials such as aluminum alloy and stainless steel, as well as typical defects such as cracks, pores, and lack of fusion. Meanwhile, complex working conditions such as torch occlusion, metal spatter, and dynamic lighting are simulated. In the preprocessing stage, an image denoising algorithm based on bilateral filtering is used to suppress arc noise, and the weld ROI region is extracted by combining adaptive threshold segmentation. For torch occlusion samples, pixel-level occlusion masks are generated by combining manual annotation with semantic segmentation models to label effective detection regions and occlusion areas. Multi-scale image pyramids are used to resample the samples, constructing a multi-resolution training set. At the same time, data augmentation strategies such as random flipping, Gaussian blur, and contrast enhancement are applied to expand sample diversity, ultimately forming a defect sample library with occlusion labels and multi-resolution annotations. This provides standardized input for the training of the multi-resolution feature fusion algorithm, and constructs the 3D coordinate mapping relationship of the weld through binocular depth information, providing geometric priors for subsequent defect localization and occlusion compensation.

2.2 Selection of object detection network

Due to the advantages of the Faster Region-based Convolutional Neural Network (R-CNN) algorithm in multi-scale feature processing, region localization accuracy, and model scalability, this paper selects it as the defect detection framework for the precision welding robot vision system. Figure 1 shows the structure diagram of the Faster R-CNN network. Through the collaborative mechanism of the Region Proposal Network (RPN) and the detection network, this algorithm can effectively generate candidate regions containing welding defects, especially small-sized cracks and lack of fusion, meeting the detection requirements of multi-scale distribution of precision welding defects. Its backbone network ResNet can extract feature maps of different resolutions, providing a natural feature pyramid structure foundation for the multi-resolution feature fusion strategy proposed in this paper, facilitating the integration of low-level edge texture information and high-level semantic information through cross-layer connections, enhancing the robust expression of defect features under interference such as complex lighting and metal spatter. In addition, the two-stage detection architecture of Faster R-CNN is superior to single-stage algorithms in localization accuracy, which can meet the sub-pixel level localization requirements of defect coordinates in precision welding. Its modular design allows customized improvements for welding scenarios, such as combining binocular vision measurement results to optimize anchor box generation strategies, or using occlusion masks to dynamically filter the occlusion region proposals generated by RPN, thereby maintaining the continuity of the detection process under torch dynamic occlusion and complex workpiece structures. This balances the dual requirements of real-time and high accuracy and adapts to the real-time response requirements of the “detection-control” closed loop in the welding robot vision system.

2.1.png

Aiming at the robust extraction demand of multi-scale defect features in the precision welding robot vision system, this paper selects the residual network ResNet50 as the basic backbone network and implements targeted improvements by introducing cross-scale residual blocks and lightweight connection mechanisms. Figure 2 shows the ResNet50 network structure diagram. Compared with the traditional VGG16 network that stacks 3×3 convolutions layer by layer, the residual connections of ResNet50 effectively solve the gradient vanishing problem in deep networks, allowing the construction of a 50-layer deep feature extraction network. It can extract features of different resolutions through the conv2~conv5 modules and enhance the noise resistance of features through the batch normalization (BN) and ReLU activation functions inside the residual blocks, adapting to complex noise environments such as arc interference and metal reflection in welding images. Specific improvements include embedding multi-branch convolution kernels in the residual blocks of the conv3~conv5 modules to enhance the ability to capture multi-directional features of slender defects like cracks. Considering the real-time requirements of welding defect detection, the number of residual blocks in the conv5 module is adjusted for lightweight design. While maintaining the high-level semantic feature extraction capability, the resolution of the feature map is increased to 1/16 of the original image, retaining more detailed information for subsequent multi-resolution feature fusion. In addition, an occlusion-aware module is introduced at the output end of the backbone network. By utilizing the pixel-level occlusion masks generated in the preprocessing stage, dynamic weighting is performed on the feature maps at conv4 and higher levels to suppress invalid feature responses in occlusion areas and strengthen the feature representation of effective detection regions.

Based on the multi-resolution feature maps output by ResNet50, this paper designs an adaptive multi-layer feature fusion strategy, combining the FPN and cross-layer attention mechanisms to achieve efficient integration of semantic information and spatial details. Figure 3 shows the FPN grid structure diagram. First, different hierarchical features are extracted through a bottom-up path: low-level conv2 features retain spatial details such as weld seam edges and textures, which are suitable for locating small-size defects; high-level conv5 features contain semantic category information of defects, used to distinguish between defect types such as cracks and lack of fusion. Second, in the top-down path, high-level features are upsampled by 2× and laterally connected to low-level features. Through weighted element-wise addition operations, the response intensity of small defects in the fused features is enhanced. For example, to address the discontinuity of crack edges in low-level features, an attention mechanism is used to assign weights to the feature maps after fusing conv2 and conv3, increasing the weight of edge features related to crack direction by 30%. In addition, to handle the problem of incomplete features caused by common local occlusions in welding scenarios, an occlusion compensation branch is introduced during the fusion process: using the three-dimensional coordinate information of the weld seam obtained from binocular vision, features of occluded areas are temporally and spatially interpolated across frames and fused with the current frame's fused features in a weighted manner, ensuring that defect features in occluded regions can still be effectively recognized. The final multi-layer fused feature map possesses both high-resolution localization capability and deep semantic expression. While maintaining detection real-time performance, the detection accuracy of small defects is improved by 25%, and the completeness recovery rate of features under occlusion scenarios exceeds 85%, meeting the real-time detection requirements of multi-scale defects in precision welding robot vision systems.

2.2.png

Figure 2. ResNet50 network structure diagram

2.3.png

Figure 3. FPN grid structure diagram

2.4 Adaptive anchor box adjustment

In conventional object detection algorithms, the predefined anchor box sizes and aspect ratios are typically derived from general-purpose datasets, making them poorly suited to the unique geometric characteristics of welding defects. To improve localization accuracy for anomalies with extreme aspect ratios, such as cracks, an adaptive anchor box adjustment strategy based on defect geometry statistics has been proposed.

The statistical method of defect geometric features was conducted on the annotated training dataset. For each defect instance, the minimum bounding rectangle was extracted, and its width and height were accurately recorded. For every bounding box, the size and aspect ratio were calculated. Specifically, for crack-type defects, the principal axis direction was also recorded. A statistical analysis was then performed on the size and aspect ratio distributions of all defects by plotting histograms to identify major distribution intervals and peaks. Furthermore, the mean, standard deviation, maximum, and minimum values of defect sizes and aspect ratios for each category were computed. Finally, for crack-type defects, a clustering analysis of aspect ratios was carried out to identify dominant aspect ratio patterns.

Aiming at the detection needs of special-shaped targets such as cracks and lack of fusion in precision welding defects, the fixed anchor box strategy of traditional RPN results in low overlap between candidate regions and real defects due to differences in preset aspect ratios, seriously affecting positioning accuracy. This paper proposes an adaptive anchor box adjustment strategy based on the GA-RPN structure. By decoupling the "position localization" and "shape prediction" in the anchor box generation process, accurate adaptation to multi-scale and unconventional-shaped defects is achieved. Specifically, under the multi-resolution feature fusion framework, using different level feature maps output by FPN, two parallel branches are constructed in the RPN stage: one branch uses low-level high-resolution features to locate the center point of the defect region, predicting the center point probability distribution through Gaussian heatmap regression, solving the positional deviation problem of dense small defects in traditional sliding windows; the other branch dynamically predicts the optimal detection box aspect ratio by combining high-level semantic features and the prior geometric database of welding defects, improving the shape matching degree between generated anchor boxes and actual defects by more than 60%. This mechanism avoids missed detection of special-shaped defects caused by fixed anchor boxes. Especially when defects are partially visible due to welding torch occlusion, candidate boxes that include the complete potential region of the defect can still be generated through joint constraints of center point localization and shape prediction. Figure 4 shows the GA-RPN network structure diagram.

2.4.png

Figure 4. GA-RPN structure diagram

2.5.png

2.5 Occlusion detection

During the welding process, the movement trajectory of the welding torch or spatter often causes 30%–70% local occlusion in the weld area, leading to feature loss in images collected by the vision system. Traditional algorithms tend to misjudge occlusion boundaries as defect edges or miss detections due to key feature loss. Therefore, this paper introduces occlusion labeling and occlusion compensation techniques to jointly address the issues of feature misjudgment and detection interruption caused by physical interferences such as dynamic occlusion of the welding torch and coverage by spatter.

To address the issue of feature loss in the detection area caused by welding torch dynamic occlusion and spatter coverage in the vision system of precision welding robots, this paper proposes an occlusion labeling strategy based on multi-level classification. By combining pixel-level semantic segmentation and manual annotation, occlusion areas in welding images are precisely labeled, and the occlusion level is divided into three grades: when the occlusion proportion of the defect target area is ≤25%, i.e., u=3, it is considered light occlusion and labeled as “partially visible”; when the occlusion proportion is between 25% and 75%, i.e., u=2, it is considered medium occlusion and labeled as “partially missing”; when the occlusion proportion is >75%, i.e., u=1, it is considered heavy occlusion and labeled as “severely occluded.” This classification strategy, combined with typical occluder forms in the welding scene, generates pixel-level masks pzz_l(u) containing information such as occlusion position, area ratio, and level, not only providing effective region identifiers for multi-resolution feature fusion but also explicitly labeling the complete contour of the occluded target, retaining the global structural prior of the defect, and providing geometric constraints for subsequent occlusion compensation. Specifically, suppose the marking information u is represented by IN(t), the t-th ground-truth bounding box is represented by HS_t, the intersection area of the t-th bounding box with all bounding boxes is represented by U_A(t), the union area of all bounding boxes in the current image is U_A, the occlusion level corresponding to the t-th bounding box is LE(t), the total number of ground-truth labels in the current image is v, and t is the t-th bounding box. Then the calculation formulas of the occlusion compensation coefficient are as follows:

$p z z_{-} l(t)=\frac{r^{L E(t)} \cdot C_{-} A(t)}{\left(r^1+r^2+r^3\right) \cdot U_{-} A}$           (1)

$C_{-} A(t)=\left|\left(H S_1 \cap H S_t\right) \cup \cdots \cup\left(H S_v \cap H S_t\right)\right|$                (2)

$U_{-} A(t)=\left|H S_1 \cup H S_2 \cup \cdots \cup H S_v\right|$               (3)

$L E(t)=\left\{\begin{array}{l}1, I N(t)= '1' \\ 2, I N(t)='2' \\ 3, I N(t)='3'\end{array}\right.$           (4)

The occlusion compensation technique relies on the contextual complementarity of multi-resolution features to achieve real-time recovery of missing features through spatiotemporal joint modeling. At the feature level, it utilizes the weld edge direction and texture details retained by low-level high-resolution features, combined with defect category information from high-level semantic features, to construct a cross-level feature interpolation model: for lightly occluded areas, missing parts are directly extrapolated using edge features from adjacent valid regions; for moderately occluded areas, 3D coordinate information acquired from binocular vision is used, and the optical flow method is employed to track feature motion trajectories in adjacent frames for spatiotemporal interpolation reconstruction; for severely occluded areas, a Generative Adversarial Network (GAN) is introduced to generate texture features consistent with the defect prior distribution. During compensation, the occlusion label level is used as a weighting factor to adjust the fusion ratio of multi-resolution features: for light occlusion, emphasis is placed on low-level detailed features; for heavy occlusion, high-level semantic guidance is strengthened, ensuring that the feature completion under different occlusion degrees conforms to visual spatial consistency and retains the essential semantic characteristics of the defect.

To enhance the model’s sensitivity to occluded areas, this paper designs a composite loss function containing the occlusion compensation coefficient pzz_l(t), integrating REP loss and IOU error loss to specifically optimize detection accuracy under occlusion scenarios. The REP loss consists of an attraction term AT and two repulsion terms RE and RB: the attraction term uses Smooth_L1 loss to force the prediction box to converge to the complete contour of the annotated box, and is weighted by the occlusion compensation coefficient, so that the model maintains localization sensitivity to defect targets under medium and heavy occlusion; the repulsion term RE uses Smooth_Ln loss to constrain the prediction box away from nearby ground-truth boxes, avoiding boundary confusion caused by occlusion, and RB reduces the dependence of the prediction box on non-maximum suppression (NMS), improving robustness in dense defect scenarios. Specifically, the occlusion compensation coefficient pzz_l(t) dynamically adjusts based on the occlusion level u of the annotation box: when u=3, a higher weight is given to the complete annotation box to enhance detail capture of partially visible defects; when u=1, the weight of background noise interference is reduced to guide the model to focus on potential defect semantics in occluded areas. The loss function is defined as follows:

$L O S S=L O S S_{I O U}+L O S S_{E R O}$              (5)

Assume the predicted confidence of the presence or absence of welding defects is represented by $\bar{Z}_u$, the ground-truth detection confidence is also represented by $\bar{Z}_u$, presence of welding defect is indicated by 1, absence by 0, the weight of IOU error is represented by η_NO, and the occlusion compensation coefficient is represented by λ(u). The expression for IOU error loss is:

$\begin{aligned} & \operatorname{LOSS}_{I O U}=\sum_{u=0}^Y m_u\left[Z_u-\lambda(u) \bar{Z}_u\right]^2 \\ & +\eta_{N O} \sum_{u=0}^Y m_u^{N O}\left(Z_u-\bar{Z}_u\right)^2\end{aligned}$              (6)

$\lambda(u)=\left\{\begin{array}{l}0, \text { Unoccluded } \\ p z z_{-} l(t), \text { Occluded }\end{array}\right.$              (7)

Let the intersection area between the predicted box O and the ground-truth box Z be denoted as IOU =AR(O∩H)/AR(O∪H), IOU =AR(O∩H)/AR(H), AR(O∩H) respectively. The union area of O and H is denoted as AR(O∪H), the matched ground-truth annotation with the predicted box is denoted by H^O_AT, and the ground-truth annotation with the largest IOU among the remaining after matching is denoted by H^O_RE. The set of all positive samples is represented as O₊=O₁∩O₂∩…∩O_|J|. δ ranges from 0 to 1. The Smooth_L1 loss and Smooth_Ln loss are denoted as SM_L1 and SM_Ln loss respectively. Then the expression for REP loss is:

$M_{R E}=M_{A T}+0.5 \times M_{R G}+0.5 \times M_{R B}$             (8)

$M_{A T}=\frac{\sum_{O \in O_{+}} S M_{L_1}\left(Y^O, H_{A T}^O\right)}{\left|O_{+}\right|}$               (9)

$M_{R E}=\frac{\sum_{O \in O_{+}} S M_{L n}\left[U P H\left(Y^O, H_{A T}^O\right)\right]}{\left|O_{+}\right|}$               (10)

$M_{R B}=\frac{\sum_{u \neq k} S M_{L n}\left[U P H\left(Y^{O_u}, Y^{O_k}\right)\right]}{\sum_{u \neq k} J\left[U P H\left(Y^{O_u}, Y^{O_k}\right)>0\right]+\epsilon}$               (11)

The expression of the Smooth regression function is as follows:

$S M_{L_1}(a)=\left\{\begin{array}{l}0.5 a^2, \text { if }|a|<1 \\ |a|-0.5, \text { others }\end{array}\right.$              (12)

$S M_{L_n}(a)=\left\{\begin{array}{l}-L N(1-a), a \leq \delta \\ \frac{a-\delta}{1-\delta}-L N(1-a), a>\delta\end{array}\right.$              (13)

2.6 Binocular vision measurement

Based on the requirement of three-dimensional spatial positioning of defects for the visual system of precision welding robots, this paper adopts binocular vision measurement technology to construct a stereo perception model of the weld area. Taking the left camera coordinate system as the origin of the world coordinate system, a binocular perspective transformation model is established by calibrating the internal parameters (focal lengths d_m, d_e) and external parameters (rotation matrix E, translation vector S) of the left and right cameras. According to the disparity principle, the coordinate difference d of spatial point O in the image coordinate systems of the left and right cameras is inversely proportional to the target depth C, i.e., c=Y*d/f, where Y is the baseline distance. By matching corresponding feature points in the left and right images, such as weld seam edge corners and defect contour feature points, and combining the sub-pixel edge information extracted by multi-resolution feature fusion, the 3D coordinates (A, B, C) of defects can be accurately calculated. This realizes the quantitative measurement of geometric parameters such as crack length and pore depth, providing precise position compensation instructions for the robot motion control module. Specifically, the world coordinate system is established with the origin p_abc of the left camera coordinate system, in which the image coordinate system is represented by p_m_A_mB_m, p_m being the projection point and the effective focal length d_m; the same applies to the right camera. The corresponding camera perspective transformation model expressions are given by the following equations:

$c\left[\begin{array}{l}A_m \\ B_m \\ 1\end{array}\right]=\left[\begin{array}{ccc}d_{m a} & 0 & z_{m a} \\ 0 & d_{m b} & z_{m b} \\ 0 & 0 & 1\end{array}\right]\left[\begin{array}{l}a \\ b \\ c\end{array}\right]$            (14)

$c_e\left[\begin{array}{l}A_e \\ B_e \\ 1\end{array}\right]=\left[\begin{array}{ccc}d_{e a} & 0 & z_{e a} \\ 0 & d_{e b} & z_{e b} \\ 0 & 0 & 1\end{array}\right]\left[\begin{array}{l}a_e \\ b_e \\ c_e\end{array}\right]$             (15)

The relative relationship between the two-coordinate systems p_abc and p_e_a_eb_ec_e is given by the following equation:

$\left[\begin{array}{c}a_e \\ b_e \\ c_e\end{array}\right]=L_{m e}\left[\begin{array}{c}a \\ b \\ c \\ 1\end{array}\right]=\left[\begin{array}{llll}e_1 & e_2 & e_3 & s_a \\ e_4 & e_5 & e_6 & s_b \\ e_7 & e_8 & e_9 & s_c\end{array}\right]\left[\begin{array}{c}a \\ b \\ c \\ 1\end{array}\right]$            (16)

The corresponding 3D coordinates of the spatial point can be expressed as:

$\left\{\begin{array}{l}a=c A_m / d_m \\ b=c B_m / d_m \\ c=\frac{d_m\left(d_m s_a-A_e s_c\right)}{A_e\left(e_7 A_m+e_8 B_m+e_9 d_m\right)+d_e\left(e_1 A_m+e_2 B_m+e_3 d_m\right)}\end{array}\right.$              (17)

In view of the feature matching error problem caused by arc light interference in welding scenarios, this paper embeds the multi-resolution feature fusion results into the binocular vision measurement process and constructs a closed-loop optimization mechanism of “feature extraction - cross-layer matching - 3D reconstruction.” At the feature extraction stage, the multi-layer feature maps output by the ResNet50 backbone network are enhanced by a cross-layer attention mechanism to improve the feature representation of low-contrast defects and enhance the robustness of binocular matching. During the disparity calculation stage, the dense block matching algorithm is applied to low-level feature maps to ensure the detail matching accuracy of small-size defects, while high-level feature maps are subjected to semantic region constraints to exclude mismatching interference from non-defect regions such as the welding torch.

Based on the actually mounted KS4A418-D binocular camera, the rotation matrix and translation vector of the left and right cameras are obtained through high-precision calibration, and an undistorted stereo vision model is constructed. During the dynamic operation of the welding robot, the 3D coordinates of the weld seam output in real-time by binocular vision are used to dynamically adjust the weight distribution of multi-resolution feature fusion: when a change in the surface curvature of the workpiece is detected, the spatial position information weight of low-level feature maps is automatically enhanced to ensure the depth measurement accuracy of defects on complex curved surfaces; in combination with the occlusion compensation module outputting occlusion masks, the 3D coordinates of the regions occluded by the welding torch are extrapolated in the spatiotemporal domain, maintaining the continuity of measurement under dynamic occlusion scenarios. This mechanism enables the binocular vision measurement system to realize real-time and accurate defect 3D coordinate solving even when the welding robot is in high-speed motion, providing a reliable spatial position reference for the “detection-control” closed loop.

This paper focused on the real-time detection requirements of precision welding robot vision systems and constructed a "data-algorithm-hardware" integrated technical framework. A specialized data acquisition device was designed to build a progressive dataset, providing multi-scale and multi-modal annotated samples for algorithm training. On the algorithm level, based on a lightweight object detection network, technical breakthroughs were achieved through a multi-resolution feature fusion system, dynamic scene adaptation mechanism, and industrial-level robustness optimization. This study overcome the bottlenecks of traditional detection algorithms in multi-scale defect representation and dynamic occlusion processing. On the theoretical level, a "feature pyramid-geometric prior-spatiotemporal compensation" detection theoretical framework was established, providing a new method for unconventional object detection. On the engineering level, the algorithm achieved high inference speed, and after integration into the welding robot vision system, the defect miss rate on the production line was effectively reduced. On the dataset level, a multi-condition dataset with 3D ground truth was built, filling the gap of standardized dynamic occlusion samples in the field of precision welding. Although the proposed method has achieved a good balance between accuracy and real-time performance, there is still room for improvement in model lightweighting and deployment efficiency. Future research is planned to follow the specific technical roadmap below: 1) Quantization target: compress the model size to ≤15MB to facilitate deployment on resource-constrained welding robot controllers or edge devices. 2) Lightweighting approach: train a more lightweight student network using knowledge distillation from the high-accuracy model optimized in this work as the teacher, significantly reducing computational cost while maintaining accuracy. Based on channel importance evaluation, redundant convolutional filters and channels in the backbone and detection head will be systematically pruned to achieve effective model compression.