Enhancing Identity Document Classification in KYC Processes: An Evaluation of the Bag-of-Visual-Words Model and Segmentation Impact

The expan the growth of online services, such as financial technology (FinTech) [1], internet banking, travel websites, and government online services [2], and more, has emphasized the importance of an efficient Know Your Customer (KYC) process. The KYC process itself is a crucial process where companies assess their potential users to ensure that their potential users can be trusted [3].

In every KYC process, a potential user or customer is expected to submit proof of their identity to the service provider [1], typically in the form of a scanned document or a photo of an official identification document. An identification document typically contains details like name, address, place and date of birth, and nationality. Some identity documents may also include additional information, such as religious affiliation [4].

From here, we can infer the necessary steps. The company must identify the type of identity document received from the user, authenticate its validity, and subsequently extract and digitize the information for further processing within the KYC procedure.

Accurate determination of the type and country of origin of identity documents can significantly aid companies in streamlining the initial phases of the KYC process. This involves ensuring that submitted identity documents align with the specified requirements and verifying their completeness. It also aids in providing a context for authenticating these documents and helps the system determine the anticipated language for Optical Character Recognition (OCR). Failure to do so could lead to unnecessary frustration for both users and service providers, leading to iterative exchanges in the KYC process, thereby causing delays or even limiting the company's capacity for customer acquisition.

In our investigation into classification approaches for identity documents, we discovered that the majority of methods rely extensively on visual features. Notably, one approach employs the Bag-of-Visual-Words (BoVW) technique [5], utilizing the SURF algorithm as a feature descriptor. Another method involves the combined usage of SURF, direct matching, and random sample consensus (RANSAC) [6].

In comparison to SIFT, SURF demands relatively fewer computational resources while producing fewer descriptors, albeit with trade-offs in lower accuracy [7, 8]. This reduction in accuracy can be attributed to several factors, one them being number of feature descriptor generated by SURF algorithm is fewer compared to SIFT. Given that identity documents inherently possess fewer distinctive features, a reduced availability of descriptors could lead to decreased model accuracy. We assert that conducting the classification process on the server side, leveraging ample resources that can be scaled as needed, mitigates the concern of conserving computational resources. Additionally, SIFT demonstrates greater resilience to noise and changes in lighting conditions [7], characteristics aligning with scenarios where customers might submit identity documents captured under various uncontrolled lighting conditions and diverse camera technologies. 

Some methods attempt to integrate visual features with other features. For example, using a histogram of gradients and color of the document, which is visual features along with the document's spatial information as a feature [9]. However, another study argues that, in the case of identity document classification, the availability of distinctive features extractable for classification purposes is limited. For instance, many identity documents share the same layout regardless of their type or issuing country [6]. Visual features can also be used along with textual features; this approach is demonstrated using SIFT to detect visual features and OCR to extract textual information [10]. However, OCR requires prior knowledge of the expected language within the document, as highlighted in the study.

Lastly, neural network based method such as CNN [11] have also been suggested to classify identity documents. Despite achieving relatively high accuracy, these methods demand extensive computational resources and substantial datasets to yield high-performing models.

Our focus on delving deeper into the BoVW method is motivated by several observations within the context of classifying identity documents for KYC processes. Firstly, the types of photos users submit are often captured under various uncontrolled lighting conditions and diverse camera technologies. Secondly, in scenarios involving sensitive information, such as processing identity documents, it is preferable to handle the process as much as possible within the company itself, rather than outsourcing, to prevent data leaks. However, the availability of samples for model training is limited. Hence, an ideal algorithm for this purpose should not require a large dataset for accurate predictions.

Aside from method selection, our investigation indicates that improving model performance often involves tuning the dataset used for training and adjusting algorithm hyperparameters. Therefore, this study explores strategies for dataset compilation, focusing on tuning hyperparameters, such as the cluster size in k-means within the proposed method. We perceive clustering descriptors into visual words as a crucial aspect of the BoVW, emphasizing the importance of finding an optimized cluster size to achieve an efficient classification model. To summarize, this study aims to explore dataset compilation strategies and the tuning of critical hyperparameters.

In addition, this study also aims to investigate how pre-processing such as segmentation, geometric transformation, and orientation affect the classification results. All pre-processing is a whole field by itself and will add a layer of complexity to the model. Pre-processing constitutes a distinct field on its own and introduces an additional layer of complexity to the model. For instance, a study has shown that imperfections in segmentation may potentially cause a model's performance to drop [5]. Fixing perspective poses a challenge by itself. First, the segmentation has to be precise; secondly, the orientation fixing might fail, causing the samples to be rejected even before the classification process [6].

We believe that our study can offer a novel perspective on dataset compilation strategies, the inclusion of pre-processing in the workflow, and how tuning cluster size can enhance the classification of identity documents.

The rest of this paper is structured as follows: In Section 2, we present related works that we have studied. In Section 3, we describe the methodology used in this study. Discussion of the results of the research that has been done is presented in Section 4. Finally, in Section 5, we draw conclusions and suggest how this research can be developed further.

In this study, we utilized the Bag-of-Visual-Words (BoVW) method for identity document classification. The BoVW model incorporates SIFT, which is used to detect descriptors, k-means clustering for grouping these descriptors, and vector quantization for constructing a visual vocabulary. The resulting visual vocabulary is then utilized for classification into pre-defined classes using an SVM (see Section 3.2). Different k values in the k-means algorithm were explored to investigate the potential improvement in model performance. Three testing approaches were used: same-variant, cross-variant, and k-fold cross-validation (see Section 3.3). Accuracy was used as the metric to assess the performance. Figure 1 shows the methodology used in this experiment.

1.png

Figure 1. Methodology

3.1 Dataset

We combined the MIDV-500 [12] and MIDV-2019 [2] datasets together for this experiment. The two datasets were chosen based on two reasons, it was available for academic research and both datasets consist of identity document images captured under various conditions, using more than one type of smartphone. The MIDV-500 dataset consisted of identity documents categorized into five sub-categories: on a cluttered background, held in hand, on the keyboard, on the table, and with partial visibility. Similarly, the MIDV-2019 dataset included identity documents categorized into two sub-categories: distorted perspective and low-light conditions.

The combined dataset consisted of a total of 21,000 samples across 50 classes, but for this experiment, only 20 classes with 8,400 samples were included. To maintain focus, identity documents issued by a single country with a single document type were excluded. Additionally, if a newer version was available, the older version was omitted. As a result, 30 classes out of the initial 50 classes were excluded, leaving 20 classes remaining for the purpose of this experiment. Table 1 displays the list of classes utilized in our experiment.

Table 1. List of identity document classes

The datasets also included the coordinates indicating the location of the identity document within the samples, which were utilized in the segmentation process. Segmentation was performed without correcting distortions like geometry transformation or rotation. After removing the background, it was replaced with black color, and the resulting image was then cropped to fit the segmented part. As a result, each image or sample obtained from the segmentation process would have a different dimension. Figure 2 shows a selection of identity document samples after the application of segmentation.

2.png

Figure 2. Examples of segmented identity documents: (a) Austrian driving license; (b) Serbian passport; (c) Croatian passport

Three datasets were generated: segmented (SGM), unsegmented (USGM), and segmented2 (SGM2). SGM consisted of samples of identity documents with segmentation, while USGM consisted of samples without segmentation. USGM served as a comparison dataset to SGM to observe if segmentation could improve accuracy. SGM2 served as a comparison dataset to SGM to assess if adding imperfections to the training dataset could enhance classification performance. These datasets were divided based on the clarity of the samples. Identity document images in the categories of clutter, hand, keyboard, and table were considered clear, where the identity documents could be seen clearly, and were grouped as the training dataset. The remaining categories, including partial, distorted, and low-light samples, were considered unclear and formed the testing dataset. Both SGM and USGM were split using the 80-20 strategy, where 80% of the samples were used for training and 20% for testing. In the case of SGM2, some samples had to be removed to maintain the dataset within the 80-20 strategy, resulting in SGM2 having a smaller sample size compared to SGM and USGM. From this point onward, the models trained with training datasets will be referred to with the suffix "-A" (e.g., SGM-A), the testing datasets with the suffix "-B" (e.g., SGM-B), and the unsplit/merged datasets without any suffix (e.g., SGM). Table 2 shows a list of our dataset variants.

Table 2. List of dataset variant

3.2 Training phase

During the training phase, all identity document samples were loaded along with their pre-defined classes. Two pre-processing steps were applied at this stage. The first step was to exclude identity document samples with low visibility, which was determined using Eq. (1). If the visibility score of a sample was below 0.1 or the identity document image's visibility was less than 10%, they were rejected and excluded from further processing. The second pre-processing step involved resizing the images to a proportional width of 320 pixels.

Visibility $=1-\frac{\text { Numbers of black pixel }}{\text { Total pixel }}$               (1)

It is important to note that since pre-defined coordinates were used to segment identity document images, we generated a list of images where the visibility score was below 0.1. This list is used in every pre-processing step to exclude images from further processing.

SIFT was utilized to detect descriptors from the identity document samples. This algorithm analyzes the difference of Gaussian (DoG), as defined in Eq. (2):

$D(x, y, \sigma)=(G(x, y, k \sigma)-G(x, y, \sigma)) * I(x, y)$               (2)

where, G(x, y, σ) represents a Gaussian function with changing scale, σ represents the scale variable of the Gaussian function, x represents the horizontal coordinate within the Gaussian window, y represents the vertical coordinate within the Gaussian window.

The equation for the Gaussian function can be defined as Eq. (3):

$G(x, y, \sigma)=\frac{1}{2 \pi \sigma^2} \exp \left[-\frac{x^2+y^2}{2 \sigma^2}\right]$               (3)

The next step involved applying the k-means algorithm to cluster the descriptors extracted from the image samples. This clustering process resulted in the generation of clusters, where each cluster represents a group of descriptors with similar characteristics. These clusters are commonly referred to as visual words or codebook entries. To explore the impact of different cluster sizes on the model performance, four distinct values were selected: 96, 128, 160, and 192. The four distinct values were selected by dividing 320, the maximum number of pixels in width that we have decided, then incrementally decrease or increase the amount by 32.

k-means clustering started by initializing centroids through randomly selecting points from the dataset. For each data point, the Euclidean distance, as defined in Eq. (4), was calculated between the data point and each of the k centroids. The data point was then assigned to the cluster associated with the nearest centroid. Subsequently, new centroids were determined for each cluster using Eq. (5). The process of recalculating distances and reassigning cluster centroids was repeated until the centroids no longer changed significantly. To reduce the complexity of the clustered visual words and achieve a more efficient representation, vector quantization was used. This quantization step assigned each local feature (SIFT descriptor) to the nearest visual word in the visual vocabulary generated by k-means clustering. As a result, the descriptor was transformed into a collection of visual words, and the histogram of visual words was generated, representing the frequency distribution of these visual words. This histogram served as a global representation of the trained samples, capturing the overall distribution of visual characteristics and patterns present in the image. In this context, the term 'visual vocabulary' simply means centroids generated by k-means clustering.

$\operatorname{dist}\left(P_1, P_2\right)=\sqrt{\left(x_2-x_1\right)^2+\left(y_2-y_1\right)^2}$               (4)

where, P₁(x₁, y₁) represents the first point, P₂(x₂, y₂) represents the second point.

$V_i=\frac{1}{c_1} \sum_{j=1}^{c_1} X_j$               (5)

where, V_i represents the centroid of cluster i, c_i represents the count or number of data points in cluster i, X_j represents the j-th data point assigned to cluster i.

In the final step of the training phase, a linear SVM was utilized to fit the generated visual words to the corresponding identity document classes. The linear SVM algorithm aimed to learn a decision boundary that effectively separated the different classes based on the visual word representations. By training the SVM on the labeled identity document samples and their associated visual word histograms, the classifier learned to distinguish between different document classes. This trained SVM model could then be used to predict the class of identity documents based on their visual word histograms.

In the testing phase of our experiment, we implemented three different strategies to evaluate the performance of our models. The first strategy, known as same-variant testing, involved utilizing each model to predict samples from the training dataset that shared the same variant as the dataset it was trained on. For instance, SGM-A was used to predict samples from SGM-B. The second strategy, cross-variant testing, required each model to predict samples from a training dataset with a different variant. For example, SGM-A was used to predict samples from USGM-B. Additionally, we performed k-fold cross-validation using the merged dataset, where k was set to 10. This involves dividing the dataset into 10 subsets or folds. In each iteration of the process, one fold was reserved for testing, while the remaining nine folds were used to train the model.

3.3 Testing phase

The testing phase consists of several key steps, including pre-processing, feature detection, vector quantization, and prediction. During the pre-processing step, images or samples with low visibility scores were excluded, and the test samples were resized to a proportional width of 320 pixels. Feature detection was performed using the SIFT algorithm. To simplify the representation of the test sample and improve efficiency, we applied vector quantization on the detected descriptors. Vector quantization assigns the test sample's descriptors to the nearest visual words in the visual vocabulary obtained during training. This step transforms the test sample into a collection of quantized descriptors, enabling compatibility with the trained model. Finally, the previously trained model, which had learned to classify the visual word representations of identity documents, was utilized to predict the class of the test sample based on its quantized descriptors. This model had been trained on labeled data during the training phase, allowing it to classify the test sample into one of the pre-defined classes.

To evaluate the performance of our models, we used accuracy as the metric. Accuracy, defined in Eq. (6), was calculated by dividing the number of correct predictions by the total number of processed samples. It is important to note that in our evaluation, true negatives and false negatives were not considered, making accuracy in our case can be considered as precision.

Accuracy $=\frac{\text { Number of correct predictions }}{\text { Total number of predictions }}$               (6)

In this paper, we have experimented with a classification approach for identity document images. The classification method involves categorizing identity documents based on their document type and the issuing country. However, one of the major challenges encountered in this task is the limited availability of discriminant features for accurate classification. Despite this challenge, the proposed method has demonstrated relatively stable accuracy on one of the dataset variants, achieving up to 97.2%.

Through our observations, we have noticed that introducing more variations in the training dataset, such as distorted, disoriented, and low-light identity documents, can enhance the performance of our model. It is important to acknowledge that the dataset used in our experiments has certain limitations. For example, every sample in a class within our dataset originates from a single identity document containing a fixed set of information. This scenario might lead to the model learning textual information as part of the image features.

To address this, future work should focus on generating variations within each class using pre-fabricated information. What this means is that research can be conducted to generate more variations of identity documents using fabricated information of an individual. This would not only increase variation in textual information but also enhance reproducibility while avoiding potential legal concerns.

Another finding indicates that by increasing cluster size in k-means we could improve model accuracy. However, it should be noted that more time and computational resources are required to build the model. Furthermore, the difference between the performance of the two upper clusters is not very significant. Additionally, we acknowledge the limitations of our research regarding the selection of cluster sizes. Which were based on a maximum width limit previously established. Therefore, further exploration of methods to optimize descriptor clustering into visual words, such as the elbow method, silhouette coefficient, and others, can be pursued.

We also discovered that refraining from performing segmentation on the training data benefits the algorithm by facilitating its learning process regarding which parts of the image are irrelevant. On the contrary, we encourage segmentation on the test data to eliminate irrelevant portions that may influence the classification of the test samples. Consequently, better methods for detecting and segmenting identity documents are still needed to enhance the overall classification process.