Intelligent Management of Public–Private Partnership Infrastructure Integrating Digital Twins and Three-Dimensional Image Reconstruction

The public–private partnership model has become the predominant framework for the development and operation of major infrastructure projects in transportation, water resources, and municipal engineering, occupying a strategically significant position within the global infrastructure landscape [1, 2]. Large-scale infrastructure undertakings are generally subject to long-term operational mechanisms, with service lives extending from twenty to thirty years. Prolonged exposure to complex natural environments and sustained traffic loading frequently induces material aging, localized damage, and performance deterioration in structural systems. The operational stability of such infrastructure is directly tied to public safety and the orderly progression of socioeconomic activities [3, 4]. Conventional infrastructure maintenance relies on periodic manual inspections and experience-based decision-making systems, which are characterized by limited detection efficiency and considerable subjectivity in evaluation criteria, rendering them inadequate for the refined and routine maintenance management demands of large-scale infrastructure clusters [5]. Digital twin technology enables the establishment of dynamic mapping relationships between physical entities and their virtual counterparts, thereby offering a novel paradigm for intelligent lifecycle control of infrastructure. Three-dimensional image reconstruction techniques, which recover geometric structures and textural features from two-dimensional visual data, constitute the core enabling technology underpinning the construction and dynamic updating of digital twin models [6, 7]. Although the deep integration of these two technological domains provides a promising pathway for intelligent infrastructure maintenance, significant deficiencies persist within this technical framework during practical engineering deployment. These limitations hinder its adaptability to the weakly textured, large-scale, and long-temporal-sequence operational scenarios characteristic of public–private partnership infrastructure, thereby substantially constraining the large-scale implementation and engineering application of intelligent management technologies [8, 9].

Substantial research progress has been achieved in the fields of three-dimensional reconstruction, digital twin updating, and intelligent defect diagnosis. Nevertheless, existing methodologies remain inadequate for meeting the high-precision, high-robustness, and automated maintenance requirements of public–private partnership infrastructure [10, 11]. In the domain of three-dimensional reconstruction, mainstream neural radiance field algorithms exhibit excellent continuous scene modeling capabilities; however, under sparse-view sampling conditions, overfitting phenomena are readily induced, preventing the precise recovery of fine-scale geometric features in weakly textured regions such as concrete and steel structures within infrastructure scenes. Conventional three-dimensional reconstruction methods, by contrast, are hampered by feature matching failures and are consequently unable to accomplish complete and accurate modeling of large-scale, open infrastructure environments [12, 13]. In the domain of digital twin model updating, existing model iteration approaches rely predominantly on light detection and ranging resampling and manual data collection, resulting in overall low automation levels and prohibitive maintenance costs. The few lightweight updating methods based on visual registration perform image matching using only single-domain spatial features, exhibiting poor adaptability to temporal environmental variations such as seasonal changes and illumination fluctuations. Consequently, cross-temporal registration accuracy undergoes substantial volatility, rendering these methods insufficient to satisfy the precision standards required for engineering applications [14, 15]. In the domain of intelligent structural diagnosis, current deep learning-based visual detection techniques enable precise localization and classification of infrastructure defects. However, only shallow visual recognition outputs are produced, without the capability to quantify three-dimensional geometric parameters of defects or to integrate with technical codes for condition rating and maintenance strategy derivation. This creates a technical gap between visual perception outcomes and engineering decision-making applications [16, 17]. Furthermore, existing research on visual infrastructure maintenance is largely confined to single visible-light data sources, with insufficient exploitation of multi-source visual information fusion. An adaptive feature fusion strategy suitable for all-weather complex maintenance scenarios has yet to be formulated, substantially limiting the environmental adaptability and scene generalization performance of intelligent maintenance systems [18, 19].

To address the aforementioned industry pain points and technical deficiencies, an integrated intelligent management framework that fuses digital twin and three-dimensional reconstruction technologies is constructed for public–private partnership infrastructure. A three-dimensional reconstruction strategy synergizing frequency-domain enhancement and geometric constraints is proposed, through which the insufficient modeling accuracy under sparse-view and weakly textured scenarios is effectively overcome. A coordinate regression network integrating dual-path frequency-domain and spatial-domain features is established, and, in conjunction with adaptive change detection and local incremental updating mechanisms, high-robustness and lightweight dynamic iteration of the digital twin model are realized. By combining three-dimensional geometric mapping with retrieval-augmented vision-language models, the complete technical chain from two-dimensional visual perception to three-dimensional semantic diagnosis is established, thereby producing standardized engineering maintenance decision outputs. Furthermore, a multi-scenario, cross-temporal annotated dataset for public–private partnership infrastructure is constructed, providing reliable data support for algorithmic iteration and performance evaluation within the research community.

The study is organized into five chapters, with a progressive and coherent logical structure throughout. In Chapter 1, the research background, existing technical limitations, and core innovations are systematically delineated, and the research value and overall framework are clarified. In Chapter 2, frontier research achievements in the field are reviewed, and the applicable boundaries and principal shortcomings of existing methodologies are defined. In Chapter 3, the overall architecture of the proposed intelligent management approach and the technical principles of each constituent module are elaborated in detail. In Chapter 4, multiple sets of comparative and ablation experiments are designed, through which the effectiveness and superiority of the proposed method are quantitatively validated. In Chapter 5, the research findings are summarized, current technical constraints are analyzed, and future research directions are discussed.

Fragmentation of technical chains is a pervasive issue in existing infrastructure digital twin operation and maintenance systems, wherein three-dimensional modeling, spatiotemporal registration, model updating, and defect diagnosis modules operate independently, and a unified and coherent automated processing workflow has yet to be established. Simultaneous satisfaction of weakly textured scene modeling accuracy, long-temporal-sequence environmental robustness, and engineering-grade semantic decision-making capabilities is unattainable through any single technical approach. Conventional solutions either rely on dense-view data and prohibitive computational costs for modeling and updating, or remain confined to the shallow perceptual level of two-dimensional visual recognition, rendering them inadequate for the long-cycle, high-precision, and unmanned refined maintenance requirements of public–private partnership infrastructure. To address the aforementioned deficiencies in the technical framework, an end-to-end four-layer progressive intelligent management architecture is constructed in this study. Driven by visual data, the construction, updating, and intelligent analysis of digital twin models are realized, thereby forming an integrated operation and maintenance technical system adapted to complex engineering scenarios.

3.1 Frequency-domain enhancement and geometry-constrained neural radiance field three-dimensional reconstruction for sparse-view weakly textured scenes

Given the significantly homogeneous textures and highly regular overall structures characteristic of public–private partnership infrastructure, combined with the engineering constraint that unmanned aerial vehicle inspection can only acquire sparse-view images, conventional neural radiance field algorithms are prone to insufficient feature representation, view overfitting, and geometric structural distortions, thereby precluding high-precision digital modeling. To overcome the aforementioned limitations, a neural radiance field reconstruction framework synergistically driven by frequency-domain optimization and geometric priors is established in this section, through which systematic algorithmic improvements are implemented across three dimensions: low-level image feature enhancement, intelligent spatial ray sampling, and model training regularization constraints. In this approach, the strong dependence of conventional spatial-domain feature modeling on rich texture information is circumvented, whereby subtle structural features of weakly textured components are extracted through frequency-domain signal analysis. Simultaneously, regularization constraints are introduced in accordance with the inherent planar and linear geometric properties of infrastructure, fundamentally improving reconstruction accuracy and structural integrity for large-scale infrastructure scenes under sparse-view conditions. Figure 1 illustrates the frequency-domain enhanced and geometry-constrained neural radiance field three-dimensional reconstruction architecture.

tu_pian_1.png

Figure 1. Architecture of frequency-domain enhanced and geometry-constrained neural radiance field three-dimensional reconstruction

A multi-scale frequency-domain adaptive enhancement strategy is first employed, through which global feature optimization is performed on the original inspection images, thereby effectively addressing the issues of sparse surface features and missing structural details in concrete and steel structures. The input spatial-domain image is assumed to have dimensions of H rows and W columns, with the corresponding image matrix denoted as I. Through two-dimensional discrete Fourier transform, the spatial-domain signal is converted into the frequency-domain signal, enabling global decoupling of structural features. The specific computational form is given as:

$F(u, v)=\sum_{x=0}^{H-1} \sum_{y=0}^{W-1} I(x, y) e^{-j 2 \pi\left(\frac{u x}{H}+v y / W\right)}$     (1)

where, u and v denote the horizontal and vertical coordinates in the frequency domain, respectively, x and y denote the pixel coordinates in the spatial domain, and F(u,v) represents the corresponding complex frequency-domain eigenvalue of the image. To achieve layered differential enhancement, the frequency-domain space is partitioned into multiple layered annular subbands, and the energy proportion of each subband is statistically computed to quantify the richness of regional features. The subband energy proportion is calculated as:

$E_s=\frac{\sum_{(u, v) \in \Omega_s}\|F(u, v)\|^2}{\sum_{(u, v)}\|F(u, v)\|^2}$    (2)

where, $E_s$ denotes the energy weight of the s-th frequency-domain subband, and $\Omega_s$ denotes the corresponding annular frequency-domain interval. Based on the energy distribution characteristics, an adaptive gain coefficient is constructed to achieve targeted enhancement of weakly textured regions:

$\lambda_s=\lambda_{\text {base }} \cdot \exp \left(-E_s / \gamma\right)$    (3)

where, $\lambda_{\text {base}}$ denotes the baseline gain magnitude, and γ denotes the gain attenuation adjustment coefficient. Higher enhancement weights are assigned to weakly textured regions with low energy proportions. After feature correction is performed through an adaptive frequency-domain filtering kernel, the optimized image is reconstructed via inverse Fourier transform, through which the concealed edge and structural details of infrastructure components are fully accentuated while the original illumination consistency is maintained.

On the basis of image feature enhancement, a frequency-guided volume sampling mechanism is designed, through which the spatial ray sampling distribution of the neural radiance field is dynamically optimized, addressing the issues of insufficient sampling of critical structures and redundant sampling in non-informative regions under sparse-view conditions. A pixel-level saliency evaluation metric is constructed by integrating edge gradient responses and local frequency-domain fluctuation characteristics, enabling precise localization of high-value structural regions. The computation formula is given as:

$Q(x, y)=\left\|\nabla^2 I^{\prime}(x, y)\right\|+\eta \cdot \operatorname{Var}\left\{F_{\text {window}}(x, y)\right\}$      (4)

where, I' denotes the frequency-domain enhanced image, $\nabla^2$ denotes the Laplacian operator, which is used to extract second-order edge features of the image, Var{ } denotes the local window frequency-domain variance, and η  denotes the dual-feature weight balancing coefficient. The pixel-level saliency value directly determines the ray sampling probability, whereby the model sampling resources are automatically concentrated on critical regions such as component edges and structural joints. The neural radiance field characterizes the three-dimensional scene through a five-dimensional mapping function, and ray color rendering is accomplished via integral operations. The core rendering formula is given as:

$\begin{gathered}C(r)=\int_{t_n}^{t f} T(t) \sigma(r(t)) c(r(t), d) d t, T(t)= \exp \left(-\int_{t_n}^t \sigma(r(s)) d s\right)\end{gathered}$     (5)

where, $r(t)$ denotes the spatial sampling ray, $t_n$ and $t_f$ denote the near and far bounds of the ray sampling interval, respectively, $\sigma$ denotes the spatial volume density, $c$ denotes the corresponding color features of the ray, and $T(t)$ denotes the ray transmittance, which is used to characterize the occlusion attenuation effect of the spatial medium. Through this sampling strategy, the overfitting problem induced by sparse inputs is effectively suppressed, and the capability of the model to infer geometric structures in unseen regions is enhanced.

To constrain the geometric plausibility of implicit modeling in accordance with the regular structural morphology characteristic of public-private partnership infrastructure, a differentiable geometric regularization loss function is introduced, through which precise constraints are imposed on the model training process. The normal vector $n_{\text {est}}$ of the fitted surface is solved via spatial density gradient computation, and the scene prior normal vector $n_{\text {prior}}$ is obtained through local plane fitting of sparse point clouds. A plane structure constraint loss is constructed to regulate the modeling accuracy of planar components such as bridge decks and utility tunnel walls:

$L_{\text {plane}}=\frac{1}{|S|} \sum_{x \in S} \arccos ^2\left(n_{\text {est}}(x) \cdot n_{\text {prior}}(x)\right)$     (6)

where, $S$ denotes the set of spatially uniformly sampled points. Through this loss function, the angular deviation between the predicted normal vectors and the prior normal vectors is quantified, whereby convex and concave distortions of planar structures are suppressed. For linear structural components such as main girders and guardrails, a linearity constraint loss is further constructed to ensure the regularity of linear structures:

$L_{\text {line}}=\frac{1}{|L|} \sum_{x \in L}\left\|\left(x-x_0\right)-vv^{\top}\left(x-x_0\right)\right\|^2$     (7)

where, $L$ denotes the set of sampling points on linear structures, $x_0$ denotes the structural reference coordinate point, and $v$ denotes the direction vector of the linear structure. Multiple constraints are integrated to construct a global loss function, through which the collaborative optimization of feature fitting and geometric regularity is achieved. The total loss expression is given as:

$L_{\text {total}}=L_{\text {rgb}}+\gamma_{\text {plane}} L_{\text {plane}}+\gamma_{\text {line}} L_{\text {line}}+\gamma_{\text {sparse }}\|\sigma\|_1$    (8)

where, $L_{r g b}$ denotes the image color reconstruction loss, and $\gamma_{\text {plane}}, \gamma_{\text {line}}$, and $\gamma_{\text {sparse}}$ denote the weight coefficients of the respective constraints. A coarse-to-fine progressive iterative training strategy is adopted. After training convergence, a high-precision triangular mesh model is extracted using the marching cubes algorithm, through which the refined reconstruction of the initial digital twin scene for public- private partnership infrastructure is ultimately accomplished.

3.2 Hierarchical scene coordinate regression network with frequency-domain and spatial-domain dual-path fusion

In response to the temporal environmental variation challenges inherent to long-term public–private partnership infrastructure maintenance scenarios, scene coordinate regression methods driven solely by single-domain spatial features are highly susceptible to illumination fluctuations, seasonal changes, and shadow variations. Feature degradation and pixel-level registration drift are readily induced, making it difficult to stably achieve high-precision mapping between two-dimensional inspection images and three-dimensional digital twin models. To address this problem, a hierarchical scene coordinate regression network with frequency-domain and spatial-domain dual-path fusion is constructed in this section, through which an end-to-end pixel-level three-dimensional coordinate prediction mechanism with strong environmental robustness is established. Figure 2 illustrates the scene coordinate regression network with frequency-domain and spatial-domain dual-path fusion. The network takes temporally new three-channel inspection images as input, and spatial prior constraints are provided by the established high-precision three-dimensional point cloud twin model. Through dual-dimensional feature encoding, adaptive feature fusion, and residual refinement regression, stable three-dimensional spatial coordinates are output pixel by pixel, providing a precise spatiotemporal alignment basis for subsequent temporal change detection and incremental model updating.

Two mutually independent and functionally complementary feature encoding branches, namely the spatial-domain and frequency-domain branches, are constructed, through which effective scene information is mined from different signal dimensions. The spatial-domain branch is built upon an improved High-Resolution Network-W48 (HRNet-W48) backbone to establish a multi-scale feature extraction structure, with four parallel resolution branches configured to perform cross-scale feature interaction and fusion, through which component contours, edge textures, and local geometric details are completely preserved, and high-precision spatial-domain feature representations are output. The frequency-domain branch focuses on eliminating feature interference caused by illumination style variations. Patch-based Fourier transform is first applied to the input images, with a patch size of 16 × 16 pixels and a sliding step of 8 pixels, whereby both the completeness of local frequency-domain information and the continuity of global features are ensured. To suppress feature distortion induced by illumination intensity shifts, standardization correction is performed on the frequency-domain magnitude spectrum. The calculation formula is given as:

$A_{\text {norm}}=\frac{A-\mu_A}{\sigma_A+\epsilon}$   (9)

where, $A$ denotes the original frequency-domain magnitude spectrum, $\mu_A$ and $\sigma_A$ denote the global mean and standard deviation of the magnitude spectrum, respectively, and $\epsilon$ denotes a small constant introduced to avoid division-by-zero issues. The correction process preserves the phase spectrum, which possesses structural invariance, while only the magnitude information, which is susceptible to illumination interference, is normalized. After inverse Fourier transform, the illumination-robust frequency-domain enhanced image is reconstructed, from which multi-scale frequency-domain features are ultimately extracted through a four-layer convolutional downsampling structure.

tu_pian_2.png

To fully integrate the high-precision advantages of spatial-domain features with the robustness advantages of frequency-domain features, a gated adaptive fusion mechanism is introduced at the four-level multi-scale features of the encoder, through which dynamic adaptive allocation of feature weights is achieved. The hierarchical feature fusion formula is given as:

$F_{\text {fusion}}^{(l)}=g^{(l)} \odot F_{\text {spatial}}^{(l)}+\left(1-g^{(l)}\right) \odot F_{\text {frequency}}^{(l)}$     (10)

where, $l$ denotes the feature scale level index, $F^{(l)_{\text {spatial}}}$ and $F^{(l)_{\text {frequency}}}$ denote the spatial-domain and frequency-domain features at the corresponding level, respectively, $g^{(l)}$ denotes the adaptive gating weight matrix, and $\odot$ denotes the elementwise multiplication operation of features. The gating weights are solved by being driven by global contextual information. Global features from both branches are aggregated through global average pooling, and weight mapping is then accomplished via a multilayer perceptron and a Sigmoid activation function:

$g^{(l)}=\operatorname{Sigmoid}\left(\operatorname{MLP}(l)\left(\left[\begin{array}{c}G A P\left(F_{\text {spatial}}^{(l)}\right) ; \\ G A P\left(F_{\text {frequency}}^{(l)}\right)\end{array}\right]\right)\right)$  (11)

where, GAP denotes the global average pooling operation, which is used to compress global feature dimensions and capture the overall properties of the scene. Through this mechanism, the feature weights of the two branches are dynamically balanced according to the real-time scene state, whereby high-precision spatial-domain features are preferentially adopted in regions with stable illumination and rich textures, while frequency-domain feature weights are adaptively strengthened in regions with drastic illumination changes and weakened textures. The network decoder employs a four-layer deconvolution structure for feature upsampling, and a residual regression structure is introduced to optimize output accuracy. The final pixel-level three-dimensional coordinates are obtained by summing coarse-grained base coordinates and refined residual coordinates, through which fitting deviations in the deep network are effectively compensated.

To regulate the network training process and ensure the geometric validity and spatial fitting accuracy of the two-dimensional-to-three-dimensional mapping, a dual-consistency joint loss function is constructed. The projection consistency loss is used to constrain the predicted coordinates to satisfy the camera imaging geometry and suppress reprojection deviations. The calculation formula is given as:

$L_{\text {proj}}=\sum_{(u, v)} \| K\left[R \mid t \widehat{X}(u, v)-[u, v, 1]^T \|^2\right.$     (12)

where, $K$ denotes the camera intrinsic matrix, $R$ and $t$ denote the camera rotation matrix and translation vector, respectively, and $\widehat{X}(u, v)$ denotes the predicted three-dimensional coordinate corresponding to pixel $(u, v)$. The model consistency loss is introduced with a Huber robust kernel function, through which the spatial distance between predicted coordinates and the reference twin point cloud is constrained, and the interference of outlier noise points is reduced:

$L_{\text {model}}=\sum_{(u, v)} \rho(\operatorname{dist}(\widehat{X}(u, v), P))$    (13)

where, dist( ) denotes the three-dimensional Euclidean distance computation, $\rho$( ) denotes the Huber robust loss function, and $P$ denotes the reference digital twin point cloud set. The total network loss is composed of the two consistency losses jointly, expressed as:

$L_{\text {regress}}=L_{\text {proj}}+L_{\text {model}}$   (14)

Through this dual-constraint formulation, network convergence is constrained from both the perspective of imaging projection plausibility and three-dimensional model fitting accuracy, whereby the stability and engineering precision of cross-temporal and cross-scene pixel-level localization are significantly enhanced.

3.3 Adaptive threshold change detection and topology-preserving incremental updating

Based on the preceding pixel-level three-dimensional coordinate mapping results, a spatial deviation characterization between temporal images and the reference digital twin model can be established, providing a quantitative basis for structural deformation and local damage detection. Based on the temporal differences of registered coordinates, a three-dimensional spatial deviation map is constructed pixel by pixel. The calculation is given as:

$\Delta(u, v)=\left\|\widehat{X}(u, v)-X_{r e f}(u, v)\right\|$     (15)

where, $\widehat{X}(u, v)$ denotes the predicted three-dimensional coordinate of the pixel in the current temporal image, $X_{\text {ref }}(u, v)$ denotes the corresponding reference three-dimensional coordinate from the reference model, and $\Delta(u, v)$ characterizes the spatial geometric offset at the pixel location. Traditional fixed-threshold discrimination methods are unable to adapt to the texture differences and structural deformation characteristics across different regions of infrastructure, readily leading to missed detections in weakly deformed regions and false detections in noisy regions. To address this issue, a local adaptive threshold discrimination mechanism is introduced, through which the discrimination criterion is dynamically updated based on the local statistical characteristics of the deviation map. Figure 3 illustrates the workflow of adaptive change detection and local topologypreserving updating. The adaptive threshold calculation formula is given as:

$\delta(u, v)=\mu_{\Delta}(u, v)+\kappa \cdot \sigma_{\Delta}(u, v)$    (16)

where, $\mu_{\Delta}(u, v)$ and $\sigma_{\Delta}(u, v)$ denote the mean and standard deviation of the deviation values within the local sliding window, respectively, which are used to characterize the deviation distribution properties of the local region, and $\kappa$ denotes the confidence adjustment coefficient, which is fixed at a value of 2.5. Based on the dynamic threshold, the pixellevel structural change probability is computed as:

$p_{\text {change}}(u, v)=\operatorname{Sigmoid}\left(\frac{\Delta(u, v)-\delta(u, v)}{\tau}\right)$    (17)

where, $\tau$ denotes the probability scaling coefficient, which is used to smooth the gradient of the probability output. To eliminate isolated noise points and ensure the spatial connectivity of change regions, a fully connected conditional random field is employed to perform global spatial smoothing on the probability map. Finally, through a binarization operation, a precise structural change mask is generated, through which the precise localization of temporal structural change regions is accomplished.

tu_pian_3.png

Figure 3. Workflow of adaptive change detection and local topology-preserving updating

On the basis of precise localization of structural change regions, a local incremental updating strategy is adopted to replace the traditional global reconstruction mode, whereby computational costs are substantially reduced while model updating accuracy is ensured. The algorithm takes the detected change mask region as the core, and a local reconstruction window is constructed by outward expansion of a fixed pixel range. Only the imaging rays corresponding to the window are selected to participate in model optimization training, effectively eliminating redundant computations in non-informative regions. During the training process, the base network weights of the initial neural radiance field model are frozen, and gradient updates are performed only for the network parameters corresponding to the local change regions, whereby the original modeling accuracy of unchanged regions is preserved to the greatest extent. To constrain the fitting quality and structural smoothness of the local update, a local joint loss function is constructed:

$L_{\text {local}}=\sum_{r \in R_{\text {window}}}\left\|C(r)-C_{\text {new}}(r)\right\|^2+\lambda_{\text {smooth}}\left\|\nabla M_{\text {update}}\right\|^2$     (18)

where, $R_{\text {window}}$ denotes the set of rays within the local reconstruction window, $C(r)$ and $C_{\text {new}}(r)$ denote the rendered color from the original model and the actual color from the new image, respectively. The first loss term is used to constrain the color fitting accuracy of the local region, and $\lambda_{\text {smooth }}$ denotes the smoothness regularization weight coefficient. The second gradient regularization term is used to constrain the spatial continuity of the updated mesh, thereby suppressing structural artifacts and geometric distortions induced by local updating.

To resolve the issues of topological discontinuity and seam distortions at the boundary between the new and old models after local updating, the Poisson surface reconstruction algorithm is introduced to achieve topology-preserving fusion of the models. Oriented point cloud data are extracted from both the locally updated mesh and the effective regions of the original reference model, and global stitching and fusion of the two point cloud sets are performed, whereby structural discontinuities and normal vector abrupt changes caused by direct stitching are avoided. By solving the Poisson implicit reconstruction equation, a globally continuous surface structure is uniformly fitted using the spatial positions and normal information of the point clouds, achieving seamless integration between the updated region and the original model region, and ensuring that the fused overall model satisfies first-order geometric continuity. Through this updating mechanism, the high-precision modeling results of non-degraded regions are completely preserved, while the model parameters of structurally deformed and damaged regions are precisely corrected. Low-cost, high-precision, and high-topological-consistency long-term dynamic iteration of the digital twin model is thereby realized, meeting the long-cycle, high-frequency, and automated maintenance updating requirements of public–private partnership infrastructure scenarios.

3.4 Multi-modal prompting and retrieval-augmented generation-enhanced defect semantic diagnosis

Instance segmentation of infrastructure surface defects is accomplished using a pre-trained DINOv2 visual encoder, through which defect instance masks, category attributes, and prediction confidences are obtained at the image level. Conventional visual detection methods can only output two-dimensional planar defect regions, without the capability to characterize the scale features of defects in physical space, rendering them insufficient for supporting structural safety assessment. Based on the pixel-level three-dimensional coordinate mapping relationship established in the preceding sections, the two-dimensional defect masks can be precisely mapped into the dynamically updated digital twin three-dimensional space, through which precise quantification of defect geometric features is achieved. Figure 4 illustrates the retrieval-augmented generation-enhanced multi-modal defect semantic diagnosis framework. By computing the three-dimensional coordinate deviations corresponding to the pixels in the defect region, the maximum extension dimension of the defect can be solved. The specific expression is given as:

$L_k=\max _{(u, v) \in R_k}\left\|\widehat{X}(u, v)-X_{r e f}(u, v)\right\|$     (19)

where, $R_k$ denotes the pixel set corresponding to the $k$-th defect instance, $\widehat{X}$ denotes the three-dimensional spatial coordinates mapped from the current image pixels, and $X_{\text {ref}}$ denotes the reference coordinates from the reference twin model. Based on the topological relationships of the matched three-dimensional point sets, the effective surface area and convex hull bounding volume of the defect region are further solved, through which a refined three-dimensional defect feature system is comprehensively constructed, encompassing spatial location, deformation scale, and damage extent, thereby providing quantifiable physical prior support for subsequent engineering semantic diagnosis.

tu_pian_4.png

Figure 4. Retrieval-augmented generation-enhanced multi-modal defect semantic diagnosis framework

 To achieve deep coupling between visual perception results and engineering domain knowledge, a hierarchical multi-modal prompting mechanism is constructed, through which a standardized input paradigm is established by integrating facility scene attributes, component types, and three-dimensional defect parameters, guiding the multi-modal model to perform specialized structural diagnosis. General-purpose vision-language models rely on parametric knowledge solidified during the pre-training phase, suffering from industry code memorization biases and insufficient adaptability to engineering scenarios, rendering them unable to produce decision content that conforms to infrastructure maintenance standards. A retrieval-augmented generation mechanism is introduced to establish a dynamic knowledge invocation pipeline. Through a dual-encoder architecture, unified vector encoding is performed on infrastructure maintenance code provisions and engineering expert cases, and a structured domain knowledge vector database is constructed. A retrieval query vector is constructed by fusing defect categories, spatial locations, and three-dimensional geometric parameters, and the matching relationship between the sample and knowledge base entries is quantified through cosine similarity. The calculation formula is given as:

$\operatorname{Sim}\left(q_k, e_j\right)=\frac{q_k \cdot e_j}{\left\|q_k\right\| \cdot\left\|e_j\right\|}$    (20)

where, $q_k$ denotes the defect feature query vector, and $e_j$ denotes the encoded vector of the $j$-th entry in the knowledge base. The top five domain knowledge entries with the highest similarity scores are selected and embedded into the prompt context, through which real-time injection of industry standards and expert experience is achieved. The limitations of generic model parameter memorization are thereby overcome, and the compliance and professionalism of defect diagnostic outputs are significantly enhanced.

A low-rank adaptation fine-tuning strategy is employed to achieve domain-specific optimization of the vision-language model. Using Qwen-VL-7B as the base model, lightweight fine-tuning is performed with a rank parameter set to 16, through which precise adaptation to the exclusive task scenario of public–private partnership infrastructure defect diagnosis is accomplished while the general cross-modal understanding capabilities of the model are preserved. Through multi-modal prompt inputs that integrate three-dimensional geometric information and retrieved knowledge, joint reasoning of visual features, spatial parameters, and domain knowledge is performed by the model, and structured diagnostic text outputs are generated. To eliminate stochastic errors in model-generated content, a post-processing verification mechanism is designed, through which automated verification and correction are performed on the physical dimension units, validity of code provisions, and condition rating criteria in the output content, ensuring the rigor and engineering usability of diagnostic results. Through this technical pipeline, the barrier between three-dimensional defect perception and intelligent engineering decision-making is effectively bridged. The conventional shallow visual recognition results are upgraded to standardized diagnostic reports that encompass three-dimensional defect parameters, damage grades, code compliance bases, and targeted maintenance recommendations, achieving an integrated closed-loop analysis of infrastructure defect perception, semantic reasoning, and intelligent decision-making.