Integrating Generative Artificial Intelligence for Robust Multi-View Registration and 3D Reconstruction of Electric Vehicles

2.1 Overall framework overview

The method proposed in this paper consists of three progressively coupled core modules. The diffusion matching matrix refinement module is responsible for establishing high-precision cross-view image correspondences. This module models matching estimation as a denoising diffusion process in the matching matrix space, utilizing epipolar geometry priors to guide multi-step reverse sampling, thereby outputting reliable feature correspondences and initial camera poses. The generative cross-view consistency enhancement module takes the matching relationships and depth estimates output by the previous module as geometric anchors, conditions a diffusion model on source-view images and depth maps to synthesize geometrically aligned virtual target-view images, and leverages contrastive learning to strengthen the cross-view discriminative power of matching features. The geometry-semantics collaborative 3D reconstruction module uses 3D Gaussian Splatting as the geometric representation carrier, receives the enhanced multi-view images and refined poses as input to complete geometric initialization, then generates novel views through a point cloud-guided video diffusion model to fill observation blind spots, and finally performs fine-grained correction on the reconstruction results using EV part semantic priors and symmetry constraints. The three modules take matching, enhancement, and reconstruction as the technical main line. The output of each preceding module is passed as differentiable input to the subsequent module, forming an end-to-end trainable technical closed loop.

The overall training adopts a strategy combining staged pre-training and joint fine-tuning. The diffusion matching matrix refinement module and the generative cross-view consistency enhancement module are first independently pre-trained with registration loss. After obtaining stable matching and feature enhancement capabilities, they are jointly optimized end-to-end with the 3D reconstruction module using rendering loss and geometric loss as supervision signals. This strategy not only ensures convergence stability of each module at the early training stage, but also achieves deep synergy between modules through subsequent joint fine-tuning, enabling generative enhancement and geometric reconstruction to mutually promote rather than operate in isolation.

2.2 Diffusion matching matrix refinement module

Image registration is the fundamental step of 3D reconstruction, and its accuracy directly determines the quality of subsequent geometric initialization. In multi-view images of EVs, metal highlights and low-texture regions cause severe degradation of the discriminative power of feature descriptors, making it difficult for traditional single-step prediction methods to directly estimate reliable matching relationships from noisy feature spaces. To this end, this paper models this problem as a denoising diffusion process in the matching matrix space, achieving coarse-to-fine correspondence estimation through multi-step iterative refinement. Figure 1 shows the structural schematic of the diffusion matching matrix refinement module.

Figure 1. Structural schematic of the diffusion matching matrix refinement module

For the input multi-view image pair $I_A, I_B \in \mathrm{R}^{H \times W \times 3}$, DINOv2 ViT-L/14 is used as a shared backbone network to extract dense visual features. This model is trained via self-supervised distillation and can maintain strong semantic discriminative power in low-texture and highlight regions. After patch processing, the images yield $N_p=H / 14 \times W / 14$ patches. Two sets of feature maps $F_A \in \mathrm{R}^{N_p \times D}$ and $F_B \in \mathrm{R}^{N_p \times D}$ are extracted, with feature dimension $D=1024$. Based on the pairwise cosine similarity between feature maps, an initial matching matrix $M^{(0)} \in \mathrm{R}^{N_p \times N_p}$ is constructed, with its elements defined as:

$m_{i j}^{(0)}=\frac{\left(F_A\right)_i \cdot\left(F_B\right)_j}{\left\|\left(F_A\right)_i\right\|_2 \cdot\left\|\left(F_B\right)_j\right\|_2}$       (1)

To ensure that the matching relationship satisfies the oneto-one mapping constraint, double softmax projection is used to regularize $M^{(0)}$ to the doubly stochastic matrix subspace $B$ $=\left\{M \in \mathrm{R}^{N p \times N p_{\geq 0}}: M 1=1, M^{\mathbb{T}} 1=1\right\}$. The projection is performed as the product of row-wise and column-wise Softmax:

$\widetilde{m}_{i j}^{(0)}=\operatorname{Softmax}_j\left(M^{(0)}\right)_{i j} \cdot \operatorname{Softmax}_i\left(M^{(0)}\right)_{i j}$           (2)

This projection makes the row and column sums of the matching matrix equal to 1, which geometrically guarantees that each image patch corresponds to only one matching patch in the other view, excluding degenerate many-to-many correspondences.

The refinement process of the matching matrix is defined as a denoising diffusion model on the doubly stochastic matrix space. The forward diffusion stage uses T = 100 steps of Gaussian noise injection:

$M^{(t)}=\sqrt{\bar{\alpha}_t} \widetilde{M}^{(0)}+\sqrt{1-\bar{\alpha}_t} \boldsymbol{\epsilon}_t, \boldsymbol{\epsilon}_t \sim N(0, I), t=1, \ldots, T$          (3)

where, $\bar{\alpha}_t=\prod_{s=1}^t\left(1-\beta_s\right)$, and the noise schedule adopts a cosine schedule:

$\beta_t=\frac{1-\cos (\pi t / T)}{2} \cdot \beta_{\max }+\frac{1+\cos (\pi t / T)}{2} \cdot \beta_{\min }$            (4)

where, $\beta_{\min }=10^{-4}$ and $\beta_{\max }=0.02$. This scheduling strategy maintains a relatively gentle noise growth in the early and middle stages, allowing the semantic structure of the matching matrix to be preserved from destruction, retaining more effective information for reverse denoising. The reverse denoising stage takes epipolar geometry priors as conditions, and gradually recovers the optimal matching matrix through the denoising network $\boldsymbol{\epsilon}_\theta$. The network adopts a 6 -layer Transformer decoder architecture, with each layer containing 8 multi-head attention heads and a 256-dimensional hidden layer, with a total parameter count of approximately 12 M. The network takes the noisy matching matrix $M^{(t)}$, the diffusion time step $t$, and the geometric condition encoding $C$ as input, and predicts the added noise:

$\widehat{\boldsymbol{\epsilon}}_t=\boldsymbol{\epsilon}_\theta\left(M^{(t)}, t, C\right)$                (5)

The geometric condition encoding $C$ is obtained by mapping the fundamental matrix $F_{A B}$ between the two views through a three-layer Multilayer Perceptron (MLP). For each candidate matching point pair $\left(p_i, q_j\right)$, its epipolar distance:

$d_{i j}=\left|p_i^T F_{A B} q_j\right| \sqrt{\left(F_{A B} p_i\right)_1^2+\left(F_{A B} p_i\right)_2^2}$              (6)

The epipolar distance d_ij is then encoded as a geometric prior feature and injected into the Transformer decoder through a cross-attention mechanism. This design enables the denoising network to continuously obtain epipolar geometric constraints during the iterative refinement process, effectively suppressing the risk of mismatching caused by falsely high feature similarity in highlight regions.

Reverse sampling follows the standard denoising diffusion iterative update:

$M^{(t-1)}=\frac{1}{\sqrt{\alpha_t}}\left(M^{(t)}-\frac{1-\alpha_t}{\sqrt{1-\bar{\alpha}_t}} \boldsymbol{\epsilon}_\theta\left(M^{(t)}, t, C\right)\right)+\sigma_t z^t$            (7)

where, $\alpha_t=1-\beta_t, \sigma_t^2=1-\alpha_{t-1} / 1-\alpha_t \beta_t, \mathrm{z} \sim N(0, I)$ when $t~>~1$, and when $t=1, z=0$ to achieve deterministic sampling. After each denoising step, double softmax projection is applied to map $M^{(t-1)}$ back to the doubly stochastic matrix subspace, ensuring that the one-to-one mapping constraint always holds during the iteration. This iterative refinement mechanism enables the matching matrix to gradually converge from a noisy state to the global optimum, effectively overcoming the inherent defect of single-step prediction that is prone to falling into local optima in weak-texture and highlight regions.

The training loss of Module 1 is a weighted combination of diffusion denoising loss and matching classification loss:

$L_{\text {match }}=E_{t, \epsilon}\left[\left\|\epsilon-\epsilon_\theta\left(M^{(t)}, t, C\right)\right\|_2^2\right]+\lambda_{\text {cls }} \cdot C E\left(M^{(0 \rightarrow T)}, M^{g t}\right)$            (8)

where, $C E(~)$ is the cross-entropy loss, used to supervise the difference between the final predicted matching matrix and the ground truth matching label $M^{g t}$, and $\lambda_{\text {cls}}$ is set to 0.1. The ground truth matching labels are generated from camera poses provided by the dataset through epipolar geometry constraints and depth consistency verification. During inference, the Denoising Diffusion Implicit Model (DDIM) deterministic sampling strategy is adopted to accelerate inference, and only $T^{\prime}=10$ sampling steps are needed to achieve accuracy comparable to $T=100$. The final output optimal matching matrix $M^*$ is further strengthened by the Sinkhorn algorithm with 20 iterations to reinforce the doubly stochastic constraint, providing high-precision camera poses and dense correspondences for the subsequent 3D reconstruction module.

2.3 Generative cross-view consistency enhancement module

The high-precision correspondences output by the diffusion matching matrix refinement module provide reliable geometric anchors for subsequent feature learning. However, the matching accuracy itself is limited by the discriminative power of feature descriptors in metal reflection and repetitive texture regions. Relying solely on original observed images to train the feature extraction network, the view coverage and lighting variations of the training data are extremely limited, making it difficult to cover the diverse imaging conditions faced by EVs in actual acquisition scenarios. To this end, this paper designs a conditional diffusion generation module $G_\phi$, which synthesizes geometrically aligned target-view virtual images conditioned on source-view images and their depth maps, thereby expanding the training distribution with virtual views and enhancing the discriminative power of features in difficult regions. Figure 2 illustrates the architecture of the generative cross-view consistency enhancement module.

Figure 2. Architecture diagram of the generative cross-view consistency enhancement module

This module takes the source-view image $I_s$ and its depth map $D_s$ as conditional inputs to synthesize the virtual aligned image $\tilde{I}_{s \rightarrow t}$ in the target view. The depth map $D_s$ is obtained by combining the matching matrix output from Module 1 with multi-view triangulation, and is smoothed by median filtering and bilateral filtering to eliminate depth noise introduced by triangulation. The generation architecture is based on the ControlNet conditional diffusion model. $I_s$ and $D_s$ are concatenated along the channel dimension and input into the ControlNet encoder to extract multi-scale conditional features $\left\{h_l\right\}_{l=1}^L$, which are injected into each layer of the U-Net decoder. Specifically, the conditional features interact with the intermediate features of the U-Net through a cross-attention mechanism, calculated as:

$\operatorname{Attention}(Q, K, V)=\operatorname{Softmax}\left(\frac{Q K^{\top}}{\sqrt{d_k}}\right) V, K=W_K h_l, V=W_V h_l$            (9)

where, $Q$ comes from the U-Net decoder features, $K$ and $V$ are obtained by mapping the conditional features through learnable projection matrices $W_K$ and $W_V$, and $d_k$ is the dimension of the attention head. This design enables the diffusion model to continuously perceive the geometric structure of the source view during the generation process, ensuring that the virtual image maintains spatial correspondence with the source view in content, rather than freely generating content unrelated to the source view.

The supervision signal for the generated image is jointly constituted by three losses. The texture consistency loss ensures that the generated image $\tilde{I}_{s \rightarrow t}$ and the real target image $I_t$ are consistent in visual appearance:

$L_{\text {texture }}=\left\|\tilde{I}_{S \rightarrow t}-I_t\right\|_1+\lambda_{\text {ssim }} \cdot\left(1-S S I M\left(\tilde{I}_{s \rightarrow t}-I_t\right)\right)+\lambda_{\text {perc }} \cdot\left\|\Phi\left(\tilde{I}_{s \rightarrow t}\right)-\Phi\left(I_t\right)\right\|_2^2$                (10)

where, $\Phi(~)$ is the Visual Geometry Group (VGG)-16 perceptual loss extraction network, $\lambda_{\text {ssim}}=0.2$, and $\lambda_{\text {perc}}=0.5$. The geometric structure preservation loss constrains the depth map of the generated image to be consistent with the targetview depth map $D_{s \rightarrow t}$ obtained by projecting the source view:

$L_{\text {geom }} \|$ Depth $\left(\tilde{I}_{s \rightarrow t}\right)-D_{s \rightarrow t}\left\|_1+\right\| \nabla \operatorname{Depth}\left(\tilde{I}_{s \rightarrow t}\right)-\nabla D_{s \rightarrow t} \|_1$             (11)

where, Depth( ) is the depth map extracted from the generated image, and $\nabla$ is the spatial gradient operator. By simultaneously constraining the depth values and their first-order gradients, this loss maintains geometric consistency at both the pixel-level accuracy and structural edge levels. The cross-view cycle consistency loss further strengthens the geometric closed loop of bidirectional generation. The generated virtual image $\tilde{I}_{s \rightarrow t}$, along with its depth map and target-view camera parameters, is input into the generator, requiring the cyclic reconstruction result to be consistent with the source-view image:

$L_{\text {cycle }}=\left\|G_\phi\left(\tilde{I}_{s \rightarrow t}, D_{s \rightarrow t}, K_t, K_s, R_{t \rightarrow s}, t_{t \rightarrow s}\right)-I_s\right\|_1$         (12)

where, $K_s$ are $K_t$ the camera intrinsic matrices of the source and target views, respectively, $R_{t \rightarrow s}$ and, $t_{t \rightarrow s}$ are the rotation and translation transformations from the target to the source view. The three losses are weighted and summed:

$L_{\text {gen}}=L_{\text {texture }}+\lambda_g L_{\text {geom }}+\lambda_c L_{\text {cycle }}$              (13)

where, $\lambda_g=0.8, \lambda_c=0.3$. This joint supervision mechanism achieves a balance between texture realism and geometric structure accuracy in the generated images, avoiding the sacrifice of geometric fidelity due to overfitting the texture loss.

A large number of generated virtual aligned image pairs are combined with the original multi-view images to construct an expanded training set. The matching feature extraction network adopts a linear attention Transformer architecture. On this basis, a cross-view hard example contrastive learning loss is introduced, forcing matching point pairs in the feature space to cluster tightly and non-matching point pairs to repel each other. For the anchor feature $f\left(p_i\right)$, the positive sample is the true matching point $f\left(q_i{ }^*\right)$, and the negative samples are all non-matching points within the same batch. A hard example mining strategy is adopted, selecting only the top $K_{\text {hard }}=10$ negative samples with the highest similarity to participate in the loss calculation:

$\begin{array}{r}L_{\text {contrast }}=-\frac{1}{B} \sum_{b=1}^B \log \frac{\exp \left(\operatorname{sim}\left(f_b^A f_b^{B+}\right) / \tau\right)}{\exp \left(\operatorname{sim}\left(f_b^A f_b^{B+}\right) / \tau\right)} +\sum_{k=1}^{K_{\text {hard }}} \exp \left(\operatorname{sim}\left(f_b^A, f_k^{B-}\right) / \tau\right)\end{array}$                  (14)

where, $B$ is the batch size, $\tau$ = 0.1 is the temperature coefficient, and ${sim}(~,~)$ is the cosine similarity. This loss is jointly optimized with the matching loss of Module 1 for the feature extraction network:

$L_{ {feat}}=L_{{match}}+\mu L_{{contrast}}$             (15)

where, $\mu$ = 0.5. The virtual aligned images play a dual role in this process: on one hand, they expand the coverage of training views, exposing the feature network to richer view variations during the training phase; on the other hand, through the construction of positive and negative samples in contrastive learning, they force the network to learn feature representations that are insensitive to highlights and repetitive textures, thereby enhancing matching robustness at its source.

2.4 Geometry-semantics collaborative 3D reconstruction module

After obtaining the enhanced multi-view images and refined camera poses, the 3D reconstruction module constructs the scene representation using the aforementioned correspondences as geometric priors. This paper adopts 3D Gaussian Splatting as the fundamental scene parameterization, which represents the target scene as a set of 3D Gaussian ellipsoids. Each Gaussian is defined by a position vector $\mu_k \in \mathrm{R}^3$, a covariance matrix $\Sigma_k \in \mathrm{R}^{3 \times 3}$, an opacity $\alpha_k \in[0,1]$, and spherical harmonic coefficients $\left\{c_{k, l} \in \mathrm{R}^3\right\}_{l=0}^{L_{s h}}$ representing view-dependent colors. In this paper, the spherical harmonic degree is set to $L_{s h}=3$. Using the high-precision matching correspondences output by Module 1, the essential matrix is estimated via the five-point method and triangulated to generate an initial sparse point cloud, which is then densified through multi-view stereo matching, obtaining approximately $5 \times 10^4$ initial point clouds $\left\{\mu^{(0)}{ }_k\right\}^{K}{k=1}$. The initial covariance of each Gaussian is set to the identity matrix scaled by 0.01 , the opacity is initialized to 0.5 , and the spherical harmonic coefficients are initialized to the mean of the corresponding view image colors. The 3D Gaussians are projected onto the 2D image plane via splatting rasterization, sorted by depth, and then alpha-composited pixel by pixel. The initial optimization objective between the rendered image $\hat{I}$ and the ground truth image $I$ is:

$L_{\text {init }}=\left(1-\lambda_{S S I M}\right)\|\hat{I}-I\|_I+\lambda_{S S I M}(1-S S I M(\hat{I}, I))+\lambda_{\text {reg }} \sum_k\left\|\Sigma_k\right\|_F^2$            (16)

where, $\lambda_{\text {SSIM}}$ controls the weight of the structural similarity loss, and $\lambda_{\text {reg}}=0.001$ is the covariance regularization coefficient, used to suppress excessive flattening of the Gaussian ellipsoids. The optimization uses the Adam solver, with a position learning rate of $1.6 \times 10^{-3}$, covariance of $5 \times 10^{-4}$, opacity of $5 \times 10^{-2}$, and spherical harmonics of $2.5 \times 10^{-3}$. Adaptive density control is triggered every 100 iterations, splitting or cloning Gaussians with excessive gradient accumulation, and pruning Gaussians with opacity below $10^{-3}$, thereby controlling Gaussian redundancy while preserving geometric details. Figure 3 shows the geometrysemantics collaborative 3D reconstruction process and correction schematic.

Figure 3. Geometry-semantics collaborative 3D reconstruction process and correction schematic