Image-Based Text Translation a Fine-Tuning Using DeepSeek-Coder and Transformer Models for Multilingual Optical Character Recognition Processing

2.1 OCR-based multilingual translation systems

The two main architectures have raised in the field of machine translation are Seq2Seq which is stand for sequence-to- sequence models and autoregressive transformers. The idea starts with each output token dependent on the one before it as autoregressive transformers create sequential outputs. So more fluent translations originated from this approach helping to capture complex dependencies in the target language while translations. A work presented by Vaswani et al. [4] about a transformer model shows this design and achieves state-of- the-art results in many translation tasks. While on the other hand systems that built using recurrent neural networks or Long Short-Term Memory units like Seq2Seq models translate the input sequence into a fixed-length context vector which is subsequently decoded into the target sequence. The fixed-size constraint of these models may make long input sequences that’s why it challenging even if they are efficient.

The need of depending on the specific needs of the work is the reason behind the choice between these architectures is based on their capacity to capture long-range connections hence autoregressive models are usually preferred.

Ouertani and Tatwany [16] developed a real-time Arabic text translation called ARx2 which is an AR mobile application. The application uses augmented reality technology to translate Arabic text to English through capturing text by the device's camera, detecting the text using the maximally stable extremal region (MSER) algorithm extracting it with ABBYY's cloud OCR finally translating it using Google cloud translate application programming interface (API). They used different dataset with numerous participant categories such as normal Arabic speakers, non-Arabic speakers visually impaired and hearing-impaired individuals to test the application. whom impressed the application's usability and efficiency in real-time translation.

A work has produced an application called AtAwAR Translate which combined a cell phone application with a mobile brain-computer interface (BCI) to develop a consumer-oriented BCI using lightweight EEG headsets and augmented reality (AR) technology which designed to evaluate the improvement accrue in the application when attention sensitivity is incorporated. The researcher highlights the potential of combining EEG-based brain interfaces with AR technology on smartphones which is one of real-world applications. Moreover, their study provides valuable insights into user preferences system usability and as the impact of attention-sensitive features on improving user interactions with AR translation applications [17].

OCR technology has seen significant improvement by deep neural networks which attaining near-human performance in reading out text in an image. And traditional techniques included rule and statistics techniques is embodied in Tesseract and that utilized HMM for character recognition. But such techniques faltered with complex script like handwriting and in most language texts. By utilizing deep neural networks in the form of Convolutional Neural Networks and Recurrent Neural Networks in an attempt to enhance performance in OCR [18].

2.2 Machine translation

A work has explored the opportunity and challenge in exploiting pre-trained Transformer architectures like GPT and BERT for language modeling tasks. It is suggested applying Coordinate Architecture Search (CAS) to augment the coarse-grained representations from pre-trained architectures and incorporate fine-grained sequence information. by using LSTM layers. Through fine-tuning of GPT and BERT's pre-trained weights, the work attempts to tailor such architectures to language modeling successfully. Experimental results show that CAS surpasses state-of-the-art language models in benchmark datasets revealing the potential of transformer architectures for NLP real world’s applications. Also mentions network transformations and the need for systematic exploration of derived networks from pre-trained architectures. Overall, the work provides valuable information to optimizing transformer architectures for language modeling applications [19].

The reference [20] addresses the role of natural language processing (NLP) in machine translation. It compares neural machine translation with NLP approaches utilized in statistical corpora, underlining the advantages of deep learning in processing high-dimensional, unlabeled, and big-data language. The paper describes the development of Google Neural Machine Translation (GNMT) and its performance in contrast with traditional phrase-based machine translation approaches. Analysis delves into the development of machine translation systems, starting with statistical approaches and moving towards neural networks, underlining the tremendous improvement in end-to-end neural machine translation post-2014. Conclusion underlines the revolution in NLP technology, specifically neural machine translation, in the translation field, supported through deep learning and big-data processing [20].

Neural machine translation took over from conventional rule-based and the phrase-based statistical translation methods are ruling multilingual text processing with its influence. By using of transformer architectures revolutionized translation performance and exploiting self-attention mechanisms with enabling a better understanding of long-range dependencies. Model architectures such as MarianMT and MBart possess high multilingual translation capability and exploiting pretrained multilingual datasets. M2M-100 has proposed a general-purpose multilingual translation system also achieving high-level performance in low-resource language pairs. In spite of these advances, transformer-based neural machine translation (NMT) models have not yet addressed difficult cases with noisy OCR, scraped text also enhancements in robustness and flexibility with respect to textual abnormalities need to be addressed [21].

2.3 OCR and machine translation

A novel multilingual scene text recognition transformer named TANGER. The main contributions of TANGER include integrating a primary vision transformer with a supplementary pyramid transformer with n-grams embeddings presenting an adaptive n-grams embedding method for flexible feature removal and designing a cross-language rectification loss function to enhance multilingual text recognition. By considering language category information and text coherence. and TANGER's adaptation process improves recognition accuracy [22].

Another work has introduced a novel method for text recognition of Arabic handwritten by using transformers. The study comparing between two end-to-end models and the transformer transducer and the standard transformer based cross-attention. And both models leverage pre-trained text and image transformers are offering non-recurrent and open-vocabulary solutions that can model complex language dependencies without relying on external language models. The proposed technique accomplishes a new advanced result on the KHATT benchmark dataset without any additional processing steps. The resulting paradoxes show that the transformer Transducer in latency so while standard transformer based cross-attention in accuracy. As a final outcome the research the is transformers in improving the accuracy and efficiency of recognizing handwritten texts in Arabic and showcased the potential of these models in developing OCR technology [23].

Another study examines how Neural Machine Translation by enhancing the traditional machine translation systems particularly in French-English translation. This analysis contrasts Deep Neural Networks with rule-based models, elucidating the advantages of Neural Machine Translation in accurately predicting word sequences. The study emphasizes that meticulous data cleaning and model evaluation which are essential and signifying the necessity for precise data processing and systematic assessment to a model's effectiveness. This paper has also argued that advancements in machine learning can enhance language translation systems [24].

Another paper has addressed some developing of end-to-end deep neural network translation system for translating English to French. There are three key tasks in the project as preprocessing model development and model run over English text. With a restricted vocabulary for enhancing training efficiency. The model performed better than older ones with an accuracy of 96.71%. In the system, tokenization and padding have been utilized for transforming text data into sequence of integers by neural networks' input. The proposed network closes the logits between neural networks and French translation via function log its_to_text. The work identifies neural networks efficient in enhancing translation machines by introducing an efficient model for translating English to French [25].

2.4 Limits in existing methodologies and the suggested mixture

The progress in OCR based multilingual translation systems has some restrictions:

1. The numerous models have difficulties with documented pictures featuring complex layouts and including tables or multiple columns with embedded graphics resulting in diminished translation accuracy.

2. Current systems frequently show suboptimal performance in handling a low resources language because of insufficient training data.

3. Inaccuracies in the OCR phase might extend into the translation output letting to exacerbating errors.

To tackle these issues this work has strategy to utilizes the DeepSeek-Coder model which is an autoregressive transformer specifically fine-tuned for OCR based translation jobs. As DeepSeek-Coder adeptly captures contextual dependencies through a sequential token creation and alleviating the effects of OCR faults and improving translation fluency.

A work using autoregressive transformers have been extensively applied to natural language generation such as for text summarization or code generation and translation. GPT-models have been seen to have high efficacy in generating coherent and fluent text through sequential prediction of single token at the time. 1.3 billion parameter deep model like DeepSeek-Coder is exhibited high generalizability in low-resource environments. Autoregressive transformers in contrast to encoder-decoders has produce text sequentially and therefore it can model long-range dependencies effectively. The outperform of conventional transformers in handling noisy text inputs and therefore are best utilized for OCR based multilingual translation workloads [25]. Machine translation evaluation is critical in deciding model performance. BLEU and METEOR scores have become prevalent for testing translation accuracy through comparisons with generated and reference translations. As the ROUGE-L and TER yield supplementary information regarding fluency and post-editing work. Perplexity is increasingly being used as a metric for translation fluency and cohesion in autoregressive models. All those measures have routine in translation studies [26].

Previous methods were mostly unable to generalize across distortions caused by OCR. And many of Seq2Seq models has the difficulties with incorrect inputs or unsegmented tokens. The misalignment between OCR outputs and subsequent translation models creates further inefficiencies. While this research addressed these deficiencies using autoregressive modeling and a meticulously regulated evaluation framework.

Recent developments in end-to-end OCR translation includes scene text translation systems like as TANGER [22] and post-OCR correction models as SMARTFEAT [8] and ARx2 [16]. and these systems address the close integration of character recognition and translation. In contrast to modular OCR then translate systems such as end-to-end models simultaneously optimize recognition and semantic transfer which providing a resilience to input noise. This research aligns with these initiatives by employing autoregressive decoding on noisy OCR data so replicating a cohesive translation experience but through distinct OCR and machine translation (MT) modules.

This work is designed to effectively extract and translate then assess multilingual text in images using OCR and deep learning-machine translation model approaches. And it is facilitated through a clearly defined pipeline by comprising preparation of datasets extraction of text via OCR by model fine-tuning and performance analysis. And it aims to assess performance of a range of transformer-based language models in translating extracted text by OCR specifically in processing noisy and incomplete datasets.

The initial part of the process entails preparation of datasets during which a group of 1,000 photos with both French and English text was compiled in the images. There is a range of textual sources in the photos including like documents and thus variation in format and structure in them. Because text produced via OCR is not perfect and always have mistakes, it is critical to develop a dataset with a range of text-related faults by including variation in fonts distortions in the text or occlusions and background noise. All of the photos underwent processing in a manner that preserved native alignment between translation in both French and English.

5.png

Figure 5. Data pre-processing

Before extraction the photos were preprocessed to improve readability for OCR is shown in Figure 5. Preprocessing is essential in the sense that OCR algorithms struggle with poor, blurry and noisy images. And the photos initially went through grayscale conversion by reducing unnecessary variations in colors that may weaken text detection. Noise filtering techniques including Gaussian blur and median filtering also were used in a bid to eliminate deformities. Adaptive thresholding is binarization was then carried out by the system and with an improvement in the differentiation of the background from the text. For an improvement in accuracy in the sense of OCR and OpenCV-based bounding box detection carried out text localization so separating areas of text from other areas in the photos. Preprocessing activities guaranteed EasyOCR high-accuracy extraction of text in the photos.

This work has assembled a bilingual image-text dataset including 1000 images including both English and French text. The images facilitating the acquisition of accurate translations. Figure 6 shows the distribution of text lengths within the photos, encompassing both are a complete paragraph so indicating that the dataset includes occurrences of varying text lengths. This diversity is essential for assessing translation models in various real-world contexts. The 1,000 images are with Arial Regular 12 pt black text on a white 800 × 200 px as PNG canvas 96 dpi integrating randomized vertical jitter ±10 px for layout variation.

6.png

Figure 6. Sample of the dataset

Following the preprocessing an extraction of text via OCR was conducted with EasyOCR which is a deep learning-powered OCR tool. EasyOCR can employs Convolutional Recurrent Neural Networks and Connectionist Temporal Classification loss which allowing it for multi-language character recognition accurately. For each image the model detected regions of text and extracted bounding boxes and transcribed detected regions into computer text. The OCR tool was trained to discriminate between languages according to position of the text in the image since the corpus consisted of English-French dual-language text. The extracted text was aggregated by a parallel corpus of English-French text pairs for training and testing translation models.

Upon completion of extraction with OCR the preparation of the dataset for fine-tuning of translation models took place. Due to the high rates of errors in extracted text with OCR including misspelling or omission of letters and wrong punctuation. A checking mechanism through manual intervention was adopted for fixing discrepancies in part of the corpus. The tokenization of the text was performed with SentencePiece subword encoding by enhancing translation performance through decomposition of a word into smaller and re-useable subword units. The corpus was partitioned into training and evaluation sets 90% for training and 10% for evaluation in the manner that assured that high-quality parallel text pairs helped in training by a specific evaluation set for testing performance.

This work has trained six transformer-based models which they are DeepSeek-Coder, Phi-1, MarianMT, MBart, M2M-100 and T5-Efficient-Tiny. All these six models were chosen for their architectures and aptitude for multilinguality in translating text. As DeepSeek-Coder and Phi-1 both use an autoregressive model that generates translation one token at a time to improving each token in relation to information that comes before it. It is an effective method for working with noisy OCR material and in which words can become truncated and must have contextual reassembly. While the following three MarianMT, MBart and M2M-100 use sequence-to-sequence this full sequence of input before producing translation. All three have boosted inference speeds but still it can have an issue with repairing distortions in OCR. The last model T5-Efficient-Tiny can be considered as low-resource lightweight transformer for producing low-resource text and therefore an efficient but less effective alternative.

The fine-tuning involved training each model with the OCR is extracted corpus in English and French. All training is performed by PyTorch and Hugging Face Transformers for compatibility with pre-trained language models. All model tokenizers were fine-tuned for increased compatibility with noisy OCR output. All models then underwent training with supervised training to map English source text with French translation through the use of cross-entropy loss as a target function. All the training involved gradient accumulation for increased training stability it uses the AdamW optimizer for efficient to update of model weights and use of a schedule for learning rates for allowing for a smooth convergence. And best checkpointing for evaluation was saved.

After fine-tuning is deployed a traditional machine translation evaluation metrics were used to evaluate the all models. BLEU is one of an evaluation metrics for model translation and reference translation n-gram overlap which was a key statistic that was used. BLEU is a common metric in studies in machine translation but it fails to include semantic similarity in addition to word correspondences. BLEU is added incorporation of synonym matching and stemming via METEOR a principal alternative and aided in enhancing translation generated via OCR evaluation. ROUGE-L, in addition the aided in sequence similarity evaluation and for post-edited requirements for converting a generated translation to its correct counterpart also TER Translation Edit Rate was utilized. Lastly perplexity is measured in an attempt to evaluate for fluency with a lower value signifying a model producing a more confident and fluent translation. To addressing translation inaccuracies produced via OCR necessitated new approaches. For enhancing translation accuracy contextual encoding aided in compensating for the impact of inaccuracies in OCR. Autoregressive techniques such as DeepSeek-Coder utilized previously produced tokens to enhance output, thus enhancing their effectiveness in processing text with noise. Besides that, Seq2Seq techniques such as MarianMT and MBart use self-attention to operations a rank relevant words in the source sequence thus enhancing translation accuracy. The techniques in post-processing were utilized to delete unnatural phrasing and correct minor inaccuracies produced via OCR thus enhancing translation accuracy. The final phase included putting the optimal model into production for real-time translation. The model was implemented in a PyTorch inference engine and where it translated text extracted by OCR in real-time (see Figure 7).

7.png

4.1 Hardware specifications and the required resources for model training

The following hardware resources were used are shown in Tables 1 and 2.

Table 1. Hardware specifications

Table 2. Required resources

4.2 Model fine-tuning procedure

Each transformer in this approach was has been fine-tuned for two epochs with the Adam optimizer as initial learning rate: 5×10^-5 with batch size of 16 and early stopping based on validation loss. Gradient accumulation was employed for memory management as well. While the training was conducted on an NVIDIA A100 GPU 40GB RAM which took approximately 4 hours for DeepSeek-Coder and proportionally less for smaller models.

While BLEU effectively captures the lexical overlap that occurs it inadequately reflects semantic accuracy but the METEOR and ROUGE-L mitigate this partially but as human evaluation could further validate fluency and adequacy. And TER measures editing effort needed to calculate as the percentage of insertions, deletions, substitutions and shifts required from system output to the reference.

All hyperparameters stayed consistent across the used models. And each model is trained via the AdamW optimizer and employing the library's default initial learning rate of approximately 5×10^–5. The batch size is configured at 10 with gradient_accumulation_steps set to 40 to emulate a bigger effective batch and gradient accumulation is activated for stability. Additional parameters like learning-rate schedule, weight decay and other similar things are held the default values of transformers to ensuring that performance variations can be traced to the designs themselves rather than to optimized hyperparameters.

Considering high accuracy of translation with DeepSeek-Coder as a number of problems experienced during investigation included a critical issue with accuracy in OCR text extraction and whose performance was largely resolution and quality-dependent. While poor resolution images frequently generated incorrect extraction of text and hence added spurious information to the training of corpus. A notable problem that was model bias and translation variation. Despite overall consistent translation produced by DeepSeek-Coder and also Phi-1 including MarianMT and MBart seemed to falter with biases incorporated in them through training over the range of datasets. Variability of manifested in high TER values that indicative of high translation errors. The translation capacities of a moderate level in cases such as M2M-100 is most exhibited unsteadiness in sentence form contributing poorly to overall translation quality.

Computational limitations also impacted model performance so much. Where the large models such as DeepSeek-Coder and Phi-1 are required, a significant GPU RAM therefore putting a constraint on extended training. On the other hand, T5-Tiny being the most miniature model that showed a lack of generalizability and therefore it showed extreme overfitting and failed to deliver accurate translation.

DeepSeek-Coder shown an inference latency about 25 tokens per second on NVIDIA A100 GPUs which requiring substantial computational resources 35GB VRAM which indicating suitability primarily for server-side rather than real-time resource-constrained deployment scenarios.

Lastly the possible data quality concerns must be addressed. The 1000-image dataset utilized in this work and although sufficient for preliminary comparisons could have been too unidimensional in terms of real-world translation environments. Median Y-coordinate-based extraction of text in images can have produced inaccuracies specifically in cases with non-conformity in text distribution in images. So those concerns point towards future improvement in terms of developing OCR extraction techniques can expansion of datasets and model fine-tuning for increased computation efficiency.

For the complete evaluation of the strengths and weaknesses of all models that are used this work has conducted a qualitative comparison of sample translations especially in examples containing frequent OCR noise like misrecognized characters for example the character "O" with number "0" or missing broken words and aberrant punctuation.

The DeepSeek-Coder model repetitively has demonstrated the ability to conclude and correct standard OCR errors such as mistaken digits or letters and missing or obscured words. As an example, when the OCR identified "Room 1O" using a "O" rather than a "0" but DeepSeek-Coder correctly rendered it as "Chambre 10" in French and inferred the correct numeral from the surrounding words. And in many cases where OCR omitted a word like “Please the form” the DeepSeek-Coder often generated a reasonable and grammar-corrected translation "Veuillez remplir le formulaire."

While the Phi-1 had robust time for inconsistent in correcting errors of this kind which also sometimes reproduced small OCR deformations or made them literal like “Room 1O” became “Chambre 1O” the “O” is character not a number.

But the models like MarianMT, MBart, and M2M-100 also found it hard to correct or recover from faulty input either by copying the errors in the output or deleting words or producing ungrammatical translations. And during a particular instance the model MarianMT translated "internat10nal" which stood for "international" as "internat10nal" in French without correcting any errors. The model T5-Tiny largely produced incorrect outputs or empty translation.

T5-Tiny is possessing barely 10M of parameters and was purposefully chosen as a low-resource baseline to underscore the significance of model capacity. The shown analysis of T5-Tiny's performance demonstrates that a model of limited size struggles to generalize to noisy OCR data and the results indicate that T5-Tiny's translations are frequently erroneous or absent and this led to that larger models such as DeepSeek-Coder and Phi-1 are more adept at handling the intricacies of real-world OCR text translation. Additionally, the T5-Tiny exemplifies the significance of model size for this task and validating the necessity for the bigger transformer models employed in this research.

The analysis was employed to determine the effect of OCR errors in the captured text for the subsequent translation quality. The OCR errors were classified into three types first uncertainty due to very small characters second errors in word segmentations and lastly lack of content.

Some small errors like single character errors were often corrected by DeepSeek-Coder and lesser so Phi-1. While Seq2seq models like MarianMT managed to carry over these errors to straight into the translation or skipped the stricken word altogether.

But the word segmentation errors due to splits or merges took by OCR generated omissions or mistranslations in all models except DeepSeek-Coder showed better and achieved accurate reconstruction of the intended meaning more often than the other models.

While in lack of substance the undetected words by OCR represent the most important problem. No model systematically retrieved missing information but the results showed the DeepSeek-Coder sometimes included reasonable generic words or paraphrased for coherence.

Quantitative analysis showed a clear correlation for those phrases having a higher density of OCR errors showed significantly lower BLEU and METEOR scores on all models and DeepSeek-Coder showed the smallest decline. This shows that the real-world limitations of available OCR trains and the importance of translation models being stable under more noisy input conditions.