Automatic Hindi OCR Error Correction Using MLM-BERT

Error correction is a necessary step to improve the accuracy of the OCR output text. This section discusses the various OCR error correction approaches used for post-processing the OCR output for Indian languages.

Post-processing techniques have been extensively used in the past to enhance OCR accuracy. Various OCR post-processing techniques are explored by Nguyen et al. [13]. These techniques can be grouped based on usage of (i) Dictionaries or Lexicon (ii) Syntactic or Semantic rules of the language/word morphology (iii) Statistical Language Models usually represented as n-grams etc., and (iv) Deep Learning approach. Some of the methods find the best alternative, while other methods find the top-n possible alternatives.

Dictionary-based approach on the Hindi OCR recognized document showed a character accuracy of 93% [14] The problem with the dictionary-based approach is that it assumes the word suggested as true if it is present in the dictionary and also it cannot correct the contextual errors.

Syntactic and semantic-based approaches for error correction can be found here. A Morphological analyzer was utilized for Bangla (Indian script) to find the root words and provide correct suggestions [15]. The corrections made here are based on a fast dictionary access method and suggest the correct word in 82.44% of cases.

The shape-based post-processing technique was applied to Gurmukhi (Indian script) and achieved a 3% improvement, attending to 97.34% character accuracy [16]. Here, consonants are categorized into groups based on shape similarity, each assigned a unique number. The input word is encoded accordingly. If the code is found in the dictionary, a matching operation is executed between the input word and words stored under that specific code. In cases without a match, structurally similar words to the input are suggested as alternatives. This approach finds difficulty in providing suggestions to similar characters.

Locality Sensitive Hashing (LSH) was used to create word clusters to improve the Telugu language OCR accuracy [17], resulting in 79.12% word accuracy. Here the OCR outputs of word images within a cluster are compared, and improvements are achieved through methods such as Character Majority Voting or Dynamic Time Warping Technique. This technique fails to provide suggestions for unique words (words appearing only once in a cluster).

The Weighted Syllable-based Spell Correction Approach (WSSC) was applied to Hindi OCR-recognized text, resulting in 88.22% word accuracy [7]. In this method, a syllable confusion matrix was generated, taking into account the frequency of each syllable substitution. Word was looked up in the dictionary to check whether it is correct or incorrect. If the word is incorrect, then based on the confusion matrix, appropriate syllables are substituted for all syllables in the words. Forming many words and assigning the weight to words based on the frequency of syllables. Again, these words were looked up in the dictionary for their correct combination, suggesting the best word based on the weight assigned. It will not suggest contextual suggestions. All these OCR post-processors implement post-processing at the Unicode (or character) level.

A graph-based, sub-character level language model was applied to rectify OCR word errors in Malayalam [18]. In this approach, edges represent the language information, and nodes represent visual similarities. The optimal path from source to destination provides suggested words. Through this approach word accuracy of 95.0% is achieved [18]. This approach fails to provide suggestions for rare words, proper nouns and foreign words.

Statistical language models (SLM) and Dictionary-based methods have been used in error correction for Indian languages such as Hindi, Punjabi (Gurmukhi script), Telugu, and Malayalam [9]. This approach reported Improvements in errors in ranges between 38% to 66% for the above languages. The limitation of this work is that a dictionary and character n-gram alone cannot effectively suggest the correct word due to the inflectional nature of Indian languages.

Sub-word embeddings have also been applied for OCR error correction in Hindi, Sanskrit, Kannada, and Malayalam, resulting in 90.42% word accuracy for Hindi [12]. Further intersection of error rectification and cluster analysis was observed in the clustering of similar words within documents for error correction, particularly in Hindi and English [8]. This approach lacks contextual suggestion.

Spelling correction in OCR-generated Hindi text, utilizing methods like Word Embedding and Levenshtein Distance [11]. This uses the Continuous Bag-of-Words (CBOW) model. It does not consider the order of words in a given context. Thus, it will struggle to provide contextual suggestions for incorrect words.

Recently, deep learning techniques were used for OCR error detection and correction spanning four languages—Hindi, Malayalam, Sanskrit, and Kannada—specifically an LSTM approach, particularly relevant when languages are written in distinct scripts [10]. It showed 92% of word accuracy for Hindi. LSTMs can capture dependencies over longer sequences than simple RNNs, but they still have a limited context window.

The above literature review indicates that post-processing of OCR output for Indian languages is challenging due to context-dependent errors and the inherent nature of highly inflectional languages. The abundance of words due to the inflectional nature of the Indian language also poses a challenge in OCR error correction.

This section discusses two main parts of the experimental setup. The first part describes how the dataset was created for testing the error correction model. The second part provides details of the experimental setup.

4.1 Test dataset creation

The procedure to create datasets to evaluate the error correction model is discussed here. Two sets of test data were utilized to evaluate the performance of the proposed error correction model. The first dataset is the Hindi OCR dataset obtained from IIITH [5]. It comprises a total of 100 pages presented in image format, along with their respective text files. For the sake of convenience, we have named this dataset as Hindi-IIITH-dataset. This dataset showed 91.6% word accuracy on OCR for Indian languages [5] and 89.29% on Tesseract Hindi OCR [7]. The second dataset is created from a part of Hindi text within the same domain, which has been utilized for training in this work. The total number of words available in the Hindi-IIITH-dataset is 29884. The characteristics of this dataset is presented in Table 2. It shows the distribution of words based on the type of syllable present in the word. The eight categories of words found in the dataset are presented below -

Words without modifier: Words with Devanagari numerals or vowels or consonants with no modifiers. e.g. मन (maNa), हलचल (HaLaCaLa), 2023.

Words with Matra: Words with Matra (Matra is a Dependent vowel sign. ◌े([e]), ◌ै([ai]), ◌ो ([o]), and ◌ौ ([au]) in Devanagari [24]) e.g. वैर (VAiRa).

Words with Eekar: Words with modifier Eekar (Eekar is a Dependent vowel sign. ि◌ ([i]) and ◌ी ([e:]) in Devanagari [24]) e.g. सीप (SIPa).

Words with Ookar: Words with modifier Ookar (Ookar is a Dependent vowel sign. ◌ु([u]) and ◌ू([u:]) in Devanagari [24]) e.g. अनुभव (aNuBhaVa).

Words with Conjunct consonant: words, which have Conjunct consonant (Conjunct consonant have more than two consonant letters [25]) e.g. पल्लव (PaLLaVa).

Words with Anusvara: Words having Anusvara (Devanagari sign ◌ं (Anusuvar. Bindi) [24]) e.g. पंप (PaMPa).

Words with Candra: Words with Candra (Devanagari sign ◌ँand letter ”Candra A” [24].) e.g. चाँद (CaMDa).

Words with multiple modifiers: Words which have more than one modifier Ookar or Anusvara or Eekar and does not belong to any of the other categories e.g. मणिका (manIKA), नितारा (NITARA).

Table 2. Characteristics of data used to create dataset to test error correction model

The process that is used to create the Hindi test dataset (Hin_test_dataset) to test proposed error correction model is discussed below:

(1) The part of Hindi text sourced from [26] containing 29884 words is saved in the file named merged_text. The merged_text document was divided into total N_text files, where each file consists of 22 lines. This splitting was done to ensure that the text could be accommodated on a standard A4-sized sheet of paper.

(2) Additionally, the N_text files underwent a conversion process to generate N_Img files in the ”.png” format. The conversion involved utilizing the Nakula font style which is of size 12pt and a resolution of 150 DPI ”X” and ”Y” dimensions [27, 28].

(3) The data generated in steps 1 and 2 was named as Hin_test_dataset having N_text and N_Img Hindi files.

(4) Next the N_Img files were given as input for Tesseract OCR (Hindi). The output of the OCR process was a collection of corresponding OCR-recognized text files, which is referred to as N_OCR files. In order to assign a label (0 for correct, 1 for incorrect) to each OCR recognized word, the original N_text and N_OCR files were aligned using Recursive Text Alignment system (RETAS) [29]. It does alignment at both the word and character levels. We utilized word-level alignment. The aligned words, i.e., the ground truth word and OCRed word, are checked. If these two words are equal, then OCRed word is assigned label 0. Otherwise, the word gets labeled 1.

The Hin_test_dataset has been created for the purpose of testing the proposed error correction approach on data within the same domain on which it is trained. The proposed error correction approach is discussed in section 4.2.

4.2 Experimental setup for error correction approach

BERT originally developed for two main tasks: next sentence prediction and MLM to predict the masked word in the sentence. The word to be predicted is masked with a token [MASK] and the model generates contextual suggestions for the masked word [30].

In this proposed error correction approach, we initially mask the incorrect word, and further, the proposed error correction approach suggests the contextually appropriate word for the masked word. Figure 1 describes the proposed system, which consists of three modules: I) Training BERT tokenizer, II) Training MLM BERT model, and III) Error Correction using MLM BERT.

A detailed explanation is provided for each phase involved in the process:

Ⅰ. Training BERT tokenizer: The BERT model requires the input data in a suitable format to obtain the corresponding embedding of the sentence. The input to the BERT model is a sentence of a fixed length. Each token was given a distinct ID when the BERT model was trained. So, in order to employ a pre-trained BERT model, each token in the input sentence must first be converted into its corresponding unique ID. Sentences are padded to achieve a fixed length for input. To obtain a unique ID for a token, a BERT tokenizer is needed. The BERT tokenizer will split the word into two subwords (token). The first token is a frequently seen word (prefix) in a corpus. Two hashes are prefixed to the second token to indicate that it is a suffix following some other subwords. Further, we can convert subwords to unique IDs using the BERT tokenizer. The BERT tokenizer for Hindi has been created using BertWordPeiceTokenizer and the respective data which is shown in Table 3.

1.png

Figure 1. Proposed hindi error correction system

Table 3. Data used for Training Dev-Hindi tokenizer and Hindi_Error_Correction_Model

Ⅱ. Training MLM BERT model: Here the BertForMaskedLM architecture is employed specifically utilizing the BERT Model known as ‘bert-base uncased’. Here we have chosen ‘bert-base uncased’ as Devanagari is an uncased script. This particular model consists of 12 layers, 768 hidden units, 12 attention heads, and a total of 110 million parameters. It was trained on lowercase English text. Despite utilizing this model, it was challenging to achieve satisfactory accuracy in correcting OCR errors in Hindi text. To support Hindi text, the BERT Dev-hindi tokenizer as created in Phase I, is used and a corpus for additional training of the pre-trained BERT model ”bert-base-uncased” is also employed, which is the same as the one used during Phase I for constructing the tokenizer. The training parameters were configured as follows: Gaussian Error Linear Unit (GELU) as hidden activation function, the maximum position embeddings were limited to 512, the attention dropout rate was set to 0.1, the number of epochs was set to 3, the instantaneous batch size per device was 8, the gradient accumulation steps were set to 1, the number of sentences was 539134 and the total number of optimization steps was 50500. The GELU was chosen as an activation function as it is a high-performing neural network activation function [31]. The choice of dropout rate and epoch configuration was influenced by a study demonstrating good performance results in the context of BERT [32]. After training the training loss was 1.54.

Table 4. Example of Hindi_Error_Correction_Model output Hindi language

Ⅲ. Error Correction using MLM BERT: In this module, a lookup dictionary is utilized to identify incorrect words and the correction model suggests the appropriate word for every incorrect word. All the incorrect words were masked using [MASK] token. The word correction was achieved by utilizing the ”Dev-Hindi” model and its corresponding tokenizer i.e. ”Dev-Hindi”. Initially, the input OCR text having masked token is tokenized using the ”Dev-Hindi” tokenizer, resulting in token IDs, and the”Dev-Hindi” model generates a set of predictions for the word in the form of token IDs and their associated probabilities. These token IDs are then converted back to words using the ”Dev-Hindi” tokenizer. Employing the Levenshtein distance for the top 6 candidate words, the similarity between the original token and the predicted token was assessed by selecting those having a similarity score of more than 0.8. Ultimately, the word with the highest probability among the suggested options is selected by the system as the most suitable word. An example is shown in Table 4. The word selected by the system can be either correct or incorrect. In some cases, the system may not provide any suggestions, and in those instances, the word remains unchanged. Examples for each of these cases are discussed in section 5. The correct words are indicated in green, the incorrect words are underlined and marked in red and the words for which no suggestions were provided are indicated in blue.