Application of Deep Learning-Based Multimodal Fusion Technology in Ancient Manuscript Image Restoration and Semantic Mining

The restoration process of traditional ancient manuscript images has significant complexity characteristics, requiring not only substantial investment of human and material resources, but also demanding that operators possess profound professional background knowledge. Compared with traditional manual restoration techniques for ancient manuscript text images, emerging deep learning-driven solutions show significant advantages in the restoration of ancient manuscript text images. Adopting neural network-based methods to automatically restore incomplete regions of images can significantly reduce restoration costs and the time cost of learning professional knowledge. At the technical implementation level, although CNN exhibit excellent performance in image recognition and object detection tasks, their performance in image generation tasks is relatively limited. In comparison, GAN [4] demonstrate significant advantages in image generation, especially in image detail reconstruction and style transfer. Specifically, for the restoration task of incomplete images, the spatial distribution Ω of the training set images can be simulated in the generator of the GAN. Based on the boundary information of the incomplete regions, real images with spatial mapping relationships are generated, and the generation results are gradually converged through iterative optimization. Simultaneously, a discriminator model is constructed to evaluate the quality of the generated images, and a loss function is used to directionally optimize the generation process, ultimately achieving high-quality image restoration results.

2.1 Ancient manuscript database construction

During the digitization of ancient manuscripts, image defects caused by age or improper preservation severely affect the readability and research value of the documents. The simulation of defective regions in ancient manuscript datasets needs to follow the damage characteristics of historical documents. We utilize the free and open resources of the National Library combined with damage simulation methods to construct the ancient manuscript dataset required for the experiments. This paper focuses on ancient books of Wuyue culture, collecting the Yuejue Shu (Song engraved edition) and Wu Yue Chunqiu (Ming manuscript) from the National Library; the Xianchun Lin'an Zhi (fragmentary volume) and Wulin Jiushi (Qing printed edition) from the Zhejiang Provincial Museum; and illustrated editions from Qiantang Yishi and Xihu Youlan Zhi in the Hangzhou Local Chronicles Collection [5]. Some examples of ancient manuscripts are shown in Figure 1 below.

Figure 1. Digital ancient manuscript resources

This study focuses on the field of ancient manuscript text restoration and faces a key challenge in the experiment: there is an urgent need to construct a paired dataset containing both damaged text images and their complete forms. However, in real scenarios, such paired data is extremely scarce, because ancient manuscripts usually only exist in a single form—either damaged or complete—making it difficult to obtain two states of the same Chinese character [6]. To break through this bottleneck, we use image technology to simulate damaged text data to alleviate the problem of data scarcity. In specific implementation, in order to accurately simulate the diversity and randomness of ancient manuscript text damage, the experiment uses the "Irregular Mask" dataset [7] for random damage simulation. This dataset contains mask images of five different ratios: 5–10%, 10–20%, 20–30%, 30–40%, and 40–50%, totaling more than 60,000 images. Their shapes, sizes, and positions all exhibit a high degree of randomness (as shown in Figure 2), which can realistically reproduce the complex characteristics of text damage. Damage simulations of different ratios have differentiated training values: small-ratio damage (5–20%) helps the model master restoration techniques for slight damage, while large-ratio damage (30–50%) strengthens the model's restoration ability for severe damage. Through systematic training, the model can gradually adapt to multi-scale damage scenarios, significantly improving restoration accuracy and robustness.

2.2.png

Figure 2. Example effect of random text damage simulation

2.2 Image restoration with deep convolutional generative adversarial network

A GAN is a deep learning model that consists of two parts: a generator and a discriminator. The goal of the generator is to generate realistic images, and the goal of the discriminator is to distinguish between images generated by the generator and real images. Through mutual competition, the generator and discriminator gradually improve the quality of the images generated by the generator. The DCGAN network structure is a variant of the GAN. It introduces a deep convolutional network structure on the basis of GAN, making the generated images of higher quality and richer in detail. Both the generator and the discriminator of DCGAN adopt a CNN structure. This project aims to use the Python programming language, the Tensorflow deep learning framework, and the DCGAN to develop an efficient and accurate image restoration system. The entire technical process is divided into four core stages:

2.2.1 Data preparation stage

First, 1000 pages of digital ancient manuscript image data are collected from the China Ancient Books Resource Database website, and multi-scale normalization processing is implemented. Zero-phase Component Analysis (ZCA) Whitening is used to eliminate data correlation [8], and contrast features are enhanced through histogram equalization [9]. To avoid the overfitting problem caused by small sample datasets, we adopt data augmentation operations [10]. The method is as follows:

Given a certain coordinate (x, y) in the original image, the data augmentation transformation operation used can be summarized as the following formula (1):

$\left[\begin{array}{l}u \\ v \\ 1\end{array}\right]=M\left[\begin{array}{l}x \\ y \\ 1\end{array}\right]$           (1)

(u, v) is the corresponding coordinate after transformation, and M is a transformation matrix collectively referred to. Through this series of image augmentation operations, the diversity of the sample dataset can be effectively increased, thereby improving the training effect and generalization ability of the deep learning model.

(1) Rotation Transformation

Taking the center point of the image as the rotation point, the transformation matrix used for rotation transformation is shown in formula (2).

$M=\left[\begin{array}{ccc}\cos \theta & -\sin \theta & 0 \\ \sin \theta & \cos \theta & 0 \\ 0 & 0 & 1\end{array}\right]$             (2)

The parameter θ in the formula represents the rotation angle. To be closer to the actual application in real scenarios, this paper ultimately chooses to limit the rotation angle range between -15 and 15 degrees. This choice is based on the common rotation angle range of objects or text in actual scenarios, so as to ensure that the augmented image samples are more in line with the actual situation. After the center rotation, blank areas often appear in the image. To deal with these blank areas, this paper chooses to use a specific color for filling. This filling method is simple and effective, and can maintain the clarity and contrast of the image, while avoiding the introduction of additional noise that would interfere with the training of the model.

(2) Perspective Transformation

Through a 3 × 3 matrix and a scale factor (0.8–1.2), scale scaling and viewpoint adjustment of the image are achieved while maintaining projective geometric invariance. Given a point coordinate (x, y) in the image, the perspective transformation maps this point to a new coordinate $\left(x^{\prime}, y^{\prime}\right)$, and the transformation process follows the following formula (3):

$\left(\begin{array}{l}x^{\prime} \\ y^{\prime} \\ w^{\prime}\end{array}\right)=\left(\begin{array}{lll}a_{11} & a_{12} & a_{13} \\ a_{21} & a_{22} & a_{23} \\ a_{31} & a_{32} & a_{33}\end{array}\right)\left(\begin{array}{l}x \\ y \\ 1\end{array}\right)$            (3)

where, $a_{i j}$ are the elements in the transformation matrix, and $w^{\prime}$ is the third coordinate in the transformed homogeneous coordinate system. To obtain the point in the Cartesian coordinate system, normalization is required through $w^{\prime}$, that is:

$x^{\prime}=\frac{x^{\prime}}{w^{\prime}}, y^{\prime}=\frac{y^{\prime}}{w^{\prime}}$              (4)

(3) Affine Transformation

In order to evaluate the anti-noise ability of the model in the image restoration task, this paper also performs affine transformation and adds controllable noise (Signal-to-Noise Ratio, SNR = 30 dB) to the image. The affine transformation formula is shown in formula (5) below, and the noise power calculation method is shown in formula (6) below:

$x^{\prime}=A x+t$           (5)

where, x is the original coordinate, A is the linear transformation matrix, and t is the translation vector.

$P_{\text {noise }}=\frac{P_{\text {signal }}}{10^{S N R / 10}}$            (6)

Then, a Gaussian noise matrix N(x, y) of the same size as the image is generated, with a mean of 0 and a variance of P_noise. In Python, this can be implemented using numpy.random.normal.

(4) Style Transfer

In order to optimize the GAN, we improve the mode collapse problem through the text style transfer task [11] and enhance the diversity generation ability of GAN. We mainly study the style transfer method based on CNN, and find some ancient manuscript text style background texture images as style pattern images. Some example images are shown in Figure 3 below. First, we initialize the synthesized image by initializing it as the content image. Then, a pre-trained CNN is selected to extract image features, and the model parameters do not need to be updated during training. The Deep Convolutional Neural Network (DCNN) extracts image features layer by layer through multiple layers. The pre-trained neural network contains 3 convolutional layers, in which the second layer outputs content features, and the first- and third-layers output style features, as shown in Figure 4 below.

2.3.png

Figure 3. Ancient manuscript style pattern images

Figure 4. Style transfer

We calculate the loss function of style transfer through forward propagation (direction of solid arrow), and iterate the model parameters through back propagation (direction of dashed arrow), that is, continuously update the synthesized image. The loss function of style transfer consists of three parts:

(1) Content Loss

Content loss makes the synthesized image close to the content image in terms of content features. The lower-layer features in the CNN describe the specific visual features of the image (i.e., texture, color, etc.), while the higher-layer features are more abstract descriptions of image content. Therefore, to compare the content similarity between two images, the similarity of their high-level features in the CNN can be compared, that is, the Euclidean distance is shown in formula (7) below.

$\mathcal{L}_{\text {content }}(\vec{p}, \vec{x}, l)=\frac{1}{2} \sum_{i, j}\left(F_{i j}^l-P_{i j}^l\right)^2$           (7)

(2) Style Loss

Style loss makes the synthesized image close to the style image in terms of style features. To compare the style similarity between two images, the similarity of their lower-layer features in the CNN can be compared. Because lower-layer features contain more local image features (i.e., spatial information is too significant), we use the Gram matrix to calculate the correlation between different response layers (Formula 8), that is, while retaining lower-layer features, the influence of image content is removed, and only the style similarity is compared.

$G_{i j}^l=\sum_k F_{i k}^l F_{j k}^l$            (8)

The style similarity calculation can be expressed by the following formulas:

$E_l=\frac{1}{4 N_l^2 M_l^2} \sum_{i, j}\left(G_{i j}^l-A_{i j}^l\right)^2$              (9)

$\mathcal{L}_{\text {style }}(\vec{a}, \vec{x})=\sum_{l=0}^L w_l E_l$            (10)

(3) Total Variation Loss

Total variation loss helps to reduce noise in the synthesized image. Total variation loss calculates the sum of absolute values of image gradients. That is, when performing smoothing processing on an image, a clear image is restored by minimizing gradient changes. Different from traditional smoothing filtering methods (such as Gaussian filtering), it can reduce noise or blur while preserving edge information, and is suitable for scenarios that require a balance between denoising and boundary preservation. Combined with content loss and style loss, style transfer is achieved by adjusting the gradient distribution of the synthesized image.

For a two-dimensional image, the total variation is defined as:

$T V(I)=\sum i, j|\nabla I(i, j)|$              (11)

where, $\nabla I(i, j)$ represents the gradient of the image at point (i, j). In practical applications, discrete approximation is usually adopted, as shown in the following formula:

$T V(I)=\sum i|\Delta x I(i)|+\sum j|\Delta y I(j)|$                (12)

where, $\Delta x$ and $\Delta y$ are the first-order differences in the horizontal and vertical directions, respectively.

Finally, when the model training ends, we output the model parameters of style transfer, and the final synthesized image is obtained.

2.2.2 Network architecture design

GAN consists of two parts: a generator and a discriminator. The task of the generator is to create new samples that look like the training data, while the discriminator is responsible for distinguishing between real data and fake data produced by the generator [12]. The two compete with each other during the training process. As iterations proceed, the generation capability of the generator gradually improves, enabling it to produce samples closer to the real data, while the discriminator also becomes better at distinguishing between real and fake, until a dynamic equilibrium is reached. We improved the original GAN in terms of network architecture by replacing the fully connected networks of the original GAN with CNN architectures in both the generator and the discriminator, thus evolving into a special kind of GAN called DCGAN. DCGAN [13] achieves high-quality image generation and restoration through a DCNN structure. The network structure diagram is shown in Figure 5 below. G is a network that generates images. It receives a random noise z and generates images from this noise, denoted as G(z). D is a discriminative network that determines whether an image is "real." Its input parameter is x, where x represents an image, and the output D(x) represents the probability that x is a real image. If it is 1, it means the image is 100% real, while an output of 0 means it is impossible for the image to be real.

Figure 5. Deep Convolutional Generative Adversarial Network (DCGAN) structure

On the basis of traditional CNN, several improvements are made to the DCGAN architecture to make the model more stable and easier to train. These improvements include: (1) Both the generator and the discriminator of DCGAN discard the pooling layers of CNN. The discriminator retains the overall CNN architecture, while the generator replaces the convolutional layers with deconvolutional layers (ConvTranspose2d). (2) Batch Normalization (BN) layers are used in both the discriminator and the generator, which helps address training problems caused by poor initialization, accelerates model training, and improves training stability. (3) In the generator, all layers except the output layer use the ReLu activation function, while the output layer uses the Tanh() activation function. In the discriminator, all layers except the output layer use the LeakyReLu activation function to prevent gradient sparsity. (4) To solve the gradient vanishing phenomenon in the model, we improved the image generation model by adding an additional generator loss on the original basis. That is, the optimized model changes to one discriminator optimization iteration followed by two generator optimization iterations. From the optimized loss variation graph, it can be seen that the gradient vanishing problem is resolved, and the iterative effect of image generation is also improved compared to before. The structure diagrams of the DCGAN generator and discriminator are shown in Figure 6 below: (a) is the generator network, which gradually upsamples to generate images; (b) is the discriminator network, which gradually extracts image features for judgment.

The main components of the generator and discriminator are compared in Table 1 below:

Figure 6. Structure diagram of Deep Convolutional Generative Adversarial Network (DCGAN) generator and discriminator

Table 1. Main layers of generator and discriminator

Loss Functions:

(1) Discriminator Loss: Consists of two parts, namely the loss of real images and the loss of generated images. The discriminator attempts to minimize the loss of real images being classified as fake and the loss of generated images being classified as real. The formula is:

$\begin{aligned} & L_D=-\mathbb{E}_{x \sim p_{\text {data }}}[\log D(x)] -\mathbb{E}_{z \sim p_z}[\log (1-D(G(z)))]\end{aligned}$          (13)

(2) Generator Loss: The generator attempts to maximize the probability that the generated images are judged as real by the discriminator. The formula is:

$L_G=-\mathbb{E}_{z \sim p_z}[\log D(G(z))]$            (14)

The objective function of GAN can be expressed as:

$\begin{aligned} & \min _G \max _D V(D, G)=\mathbb{E}_{x \sim p_{\text {data }}(x)}[\log D(x)] +\mathbb{E}_{z \sim p_z(z)}[\log (1-D(G(z)))]\end{aligned}$         (15)

where, G represents the generator network, D represents the discriminator network, $p_{ {data}}$ represents the real data distribution, and $p_z$ represents the noise distribution.

For the ancient manuscript text content restoration task in this paper, we replace the input random noise z of the generator with the damaged content y, that is, the fragments of ancient manuscript images that have undergone time erosion, physical damage, or human destruction are directly used as the input source of the generator. This design has three technical advantages: First, by preserving the damaged features of the original image (such as faded textures, broken strokes, etc.), the generator can learn more targeted restoration patterns. Second, using y as input avoids the uncontrollability introduced by random noise, ensuring a high degree of consistency between the restoration results and the original ancient manuscript content in terms of style and brushstrokes. Finally, this "repairing through damage" mechanism enables the model to better handle the complex degradation patterns unique to ancient manuscripts (such as wormhole perforations, water stain bleeding, etc.). Compared with traditional noise input schemes, this method improves the text recognition accuracy. Its input paradigm essentially reconstructs the generator as a mapping function from "damage to content". Therefore, the objective function in formula (15) is adjusted to:

$\begin{aligned} & \min _G \max _D V(D, G)=\mathbb{E}_{x \sim p_{\text {data }}(x)}[\log D(x)] +\mathbb{E}_{y \sim p_{\text {damaged }}(y)}[\log (1-D(G(y)))]\end{aligned}$              (16)

2.2.3 Innovation of damage input mechanism

A Damage-Aware Module (DAM) is added to the DCGAN, which automatically distinguishes between ink fading and physical damage areas through residual learning. Its core composition and implementation are as follows:

I. Module Architecture Design

The module adopts a dual-branch residual structure, consisting of two parallel paths: (a) Texture Degradation Branch: Lightweight convolutional layers are used to extract the gradual features of ink fading areas, and the continuity of color attenuation is captured through a local attention mechanism. (b) Structural Damage Branch: Dilated convolution is used to expand the receptive field, detect the edge fracture features of physical damage, and combine gradient magnitude thresholding to enhance the response of abnormal areas. The outputs of the two branches are fused through a residual connection to form a damage mask. For dynamic weight allocation, a learnable gating unit is introduced, which dynamically adjusts the contribution weights of the two branches according to the damage type of the input image. The gating value is generated through the Sigmoid function, and the formula is:

$G=\sigma\left(W_g \cdot\left[F_{\text {texture }} \oplus F_{\text {structure }}\right]\right)$         (17)

where, F_texture and F_structure are the features of the two branches respectively, and W_g is the trainable parameter.

II. Residual Learning Mechanism

(a) Degradation Feature Decoupling: In the encoding stage of the generator, DAM separates the features of ink fading (low-frequency degradation) and physical damage (high-frequency anomaly) through residual learning:

$F_{\text {damage }}=F_{\text {input }}-F_{\text {reconstructed }}$             (18)

where, F_{reconstructed} is generated by the preliminary restoration result of the generator, and the residual F_damage is binarized as the damage mask.

(b) Adversarial Loss Constraint: A damage-aware branch is added to the discriminator, and its loss function is:

$L_{D A M}=\lambda_1 L_{a d v}+\lambda_2 L_{r e c}+\lambda_3 L_{s e g}$           (19)

where, L_seg is the semantic segmentation loss based on U-Net, which forces the generator to preserve the semantic consistency of undamaged areas during restoration.

2.2.4 Restoration effect experiment

In this experiment, we adopted the TensorFlow model framework, and used NVIDIA GPU Titan X as the training GPU to run the training algorithm of the adversarial generative network. The hyperparameters of the neural network are related to the effect of the neural network. In the process of adjusting the training of the neural network model, different hyperparameters have different effects on the training of the neural network. The main hyperparameters include: learning rate, number of neural network layers, number of hidden layer neurons, number of learning iterations, batch-size, neuron activation function, and weight initialization parameters. In this paper, the learning rate of the neural network is configured as 0.001, the lambda parameter is 0.0001, the number of iterations is 4000, and the batch-size is 100.

2.7.png

Figure 7. Example of text image restoration effect

After completing the restoration of ancient manuscript text images, the restoration results not only restore the visual integrity of the original images, but also provide significantly optimized input data for subsequent semantic mining tasks, effectively eliminating the image degradation problems caused by age, and making the detailed features of text and patterns clearer and more distinguishable. In the restored ancient manuscript pages, the edges of text strokes are sharper, and the contours of patterns are more continuous, which lays a solid foundation for text recognition (OCR) and pattern classification tasks in semantic mining. In addition, the restoration process retains the color and texture features of the original images, avoiding semantic information loss caused by excessive processing, and ensuring that the semantic mining model can extract deep knowledge based on real and reliable input data. Therefore, the restored images not only improve visual quality, but also create favorable conditions for subsequent tasks such as semantic feature extraction and historical background association analysis by providing high-precision semantic carriers.

2.2.5 Optimization of deep convolutional generative adversarial network combined with transformer architecture

From the above experimental results, it can be seen that the DCGAN has significant advantages in ancient manuscript image text restoration. The generator network can automatically learn the stroke features and texture patterns of ancient manuscript text through multi-layer convolution operations. The adversarial training mechanism can effectively restore the semantic coherence of defective regions, and its end-to-end nature enables direct restoration without complex feature engineering. However, by examining some text restoration effects, it is found that there are still some defects: (1) The limited local receptive field of the convolutional network restricts the modeling ability of long-range dependencies, leading to stroke fractures when repairing complex glyph structures; (2) The instability of adversarial training may produce artifact interference; (3) Insufficient generalization ability for small samples. To address the above problems, this study proposes to introduce the Transformer architecture [14] for modular optimization. A multi-head self-attention mechanism is embedded in the generator of DCGAN, enabling the network to capture cross-region stroke association features; at the same time, positional encoding is used to replace traditional convolution to better preserve the spatial topology of text; multi-scale feature fusion is achieved through hierarchical Transformer stacking.

The advantage of the DCGAN+Transformer restoration model is that DCGAN is good at capturing local features and texture details and can generate more realistic details; while Transformer, through the self-attention mechanism [15-18], can more accurately capture long-range dependencies, better understand contextual semantics and structural requirements, and compensate for the instability defect of DCGAN adversarial training.

The core of Transformer is the self-attention mechanism, and its calculation process is:

$\operatorname{Attention}(Q, K, V)=$ soft $\max \left(\frac{Q K^T}{\sqrt{d_k}}\right) V$         (20)

where, Q is the query matrix, K is the key matrix, V$~$is the value matrix, and $d_k$ is the dimension of the key vector.

Multi-head attention concatenates the structure of multiple attention heads:

$MultiHead (Q, K, V)=Concat \left(\right.head _1, \cdots, head \left._h\right) W^O$            (21)

The calculation of each attention head is:

$head_i={Attention}\left(Q W_i^Q, K W_i^K, V W_i^V\right)$              (22)

In order to improve the model's understanding ability of complex semantics, we innovatively adopt a cross-modal fusion mechanism. Cross-modal fusion integrates data from different modalities (such as text and image), and utilizes the parallel computing capability and long-range dependency capturing characteristics of the Transformer architecture to achieve effective interaction of multimodal information. By calculating the correlation weights between features of different modalities, the mechanism dynamically adjusts the information fusion strategy, breaks through the locality limitation of a single modality to achieve global semantic modeling, and enhances system robustness, showing strong adaptability to noise and missing data, which is especially suitable for handling degradation problems common in ancient manuscripts such as blur and damage. The module composition and specific implementation method of the cross-modal fusion mechanism innovation are as follows:

I. Dual-stream Transformer Architecture

(a) Image Stream Branch: Adopting the Vision Transformer (ViT) architecture, multi-layer self-attention mechanism is used to perform block encoding on ancient manuscript images, effectively extracting spatial features such as text morphology and binding patterns. Through a local-global attention mechanism, this branch simultaneously captures the local detail features and global structural information of ancient manuscript images.

(b) Text Stream Branch: Based on a pre-trained ancient language model (such as a Bidirectional Encoder Representations from Transformers (BERT) variant), bidirectional Transformer encoder is used to perform deep semantic modeling on ancient manuscript transcriptions. This branch can capture vocabulary associations, grammatical structures, and cultural semantics in historical contexts, generating text semantic vectors with domain adaptability.

(c) Cross-modal Interaction Layer: Embedded in the Transformer block, it dynamically calculates the correlation weights between image features and text semantics to achieve mutual correction of image and text features. Specifically, when the image stream detects blurred or missing text regions, the text stream can provide candidate completion suggestions based on the historical knowledge base, thereby guiding the image restoration process.

Specific implementation methods: (a) Feature Alignment Mechanism: In the cross-attention layer, image features are used as Value, text semantics as Query, and an attention weight map is generated through matrix operation. This mechanism forces the image restoration result to be consistent with the transcription semantics, ensuring that the restored content conforms to the historical context. (b) Dynamic Correction Strategy: An adversarial loss function is introduced during training, and the discriminator constrains the generated results. This strategy enables the generator to prioritize the preservation of key visual regions strongly related to the text stream (such as titles, seals, etc.) during restoration, while effectively suppressing the generation of irrelevant textures, thereby improving the cultural rationality and visual authenticity of the restoration results.

II. Semantic Constraint Loss Function

Module composition: (a) Historical Context Knowledge Base: Constructing a structured database containing dynasties, ritual systems, and terminology as a reference standard for semantic constraints. (b) Semantic Similarity Calculation Module: Based on cosine similarity or contrastive learning, quantifying the matching degree between the restoration result and the knowledge base. Specific implementation methods:

(a) Loss Function Design:

$L_{\text {sem }}=\lambda \cdot \operatorname{Sim}\left(F_{\text {text}}, F_{\text {image}}\right)+\mu \cdot \operatorname{KL}\left(P_{\text {hist}} / / P_{\text {pred}}\right)$            (23)

where, Sim is the similarity between image and text features, and KL is the probability divergence between the restoration result and the historical context distribution.

(b) Training Optimization: A semantic branch is added to the discriminator of DCGAN, and gradient backpropagation is used to force the generator to output images that conform to the historical context.

2.2.6 Experiment and analysis

I. Evaluation Metrics

Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index (SSIM) are two commonly used metrics in image quality assessment, which are used to measure the similarity and quality between a reconstructed image and a reference image (usually the original image).

(a) Peak Signal-to-Noise Ratio

PSNR evaluates image quality by measuring the error between the reconstructed image and the reference image. It is based on the Mean Squared Error (MSE), which is the average of the squared differences of each pixel value. The definition is as follows:

$P S N R=10 \cdot \log _{10}\left(\frac{M A X^2}{M S E}\right)$            (24)

where, MAX is the maximum possible pixel value of the image (for example, for an 8-bit image, MAX = 255). MSE is the mean squared error between the reconstructed image and the reference image. The larger the PSNR value, the smaller the difference between the reconstructed image and the reference image, and the better the image quality. The advantage of this metric is that it is simple and easy to use, with low computational cost. The disadvantage is that it only focuses on the absolute pixel difference and cannot well reflect the perceptual quality of the image, because the human eye is more sensitive to certain image features.

(b) Structural Similarity Index

SSIM is mainly used to evaluate the perceptual similarity between images, with special focus on the similarity of luminance, contrast, and structural information of the images. The SSIM formula is as follows:

$\operatorname{SSIM}(x, y)=\frac{\left(2 \mu_x \mu_y+C_1\right)\left(2 \sigma_{x y}+C_2\right)}{\left(\mu_x^2+\mu_y^2+C_1\right)\left(\sigma_x^2+\sigma_y^2+C_2\right)}$               (25)

where, $\mu_x$ and $\mu_y$ are the average luminance of the two image blocks, respectively. $\sigma_x^2$ and $\sigma_y^2$ are the contrast (variance) of the two images, respectively. $\sigma_{x y}$ is the covariance of the two images, measuring their structural similarity. C₁ and C₂ are constants used to avoid zero in the denominator. SSIM is superior to PSNR in simulating the human eye's perception of image structure; the larger the value, the better the effect. It not only focuses on the difference between pixels, but also takes into account the local structure, luminance, and contrast of the image. The advantage is that it is more consistent with the human visual system and can reflect the perceptual image quality. The disadvantage is that the computational complexity is higher, and it is more time-consuming compared to PSNR.

II. Experimental Results

This paper conducts evaluation experiments on the collected ancient manuscript text dataset. After the preprocessing operations in Chapter 2, the dataset collected PDF files of various writing styles including regular script, clerical script, Song typeface, running script, etc., and based on these, ancient manuscript single-character images were produced, containing 1200 categories and covering multiple font styles. During the training process, multiple data augmentation strategies were adopted to further improve the generalization ability of the model. In the process of verification experiments, the dataset was divided into training set and test set in a ratio of 7:3, ensuring the scientificity and rationality of the experiments. During the experiment, random damaged masks were used to perform operations with text images to simulate the damage conditions of ancient Chinese characters.

Table 2. Peak Signal-to-Noise Ratio (PSNR) evaluation of ancient manuscript text image restoration under different methods

Table 3. Structural Similarity Index (SSIM) evaluation of ancient manuscript text image restoration under different methods

Experiments show that this DCGAN+Transformer hybrid architecture outperforms pure DCGAN or pure Transformer models in both PSNR and SSIM metrics, especially showing significant advantages in complex texture restoration tasks. The experimental data and comparisons of various methods are shown in Table 2 and Table 3.