Influence of Class Imbalance and Resampling on Classification Accuracy of Chronic Kidney Disease Detection

The objective of chronic kidney disease prediction is to identify whether a person is suffering from kidney disease by classifying a given observation into two classes: patient class (positive class) and non-patient class (negative class). To detect chronic kidney disease, the predictive model uses chronic kidney disease features to characterize each observation in the dataset. Automated predictive models are important in decision making especially in giving an early indication of whether a person should be tested for chronic kidney disease or not. Automated test results are used in deciding whether further tests and additional screening is required or not. If the automated test using the predictive model is positive, the patient will be subjected to additional tests, but no further test will be recommended afterwards. The emphasis, however, is not only on which class a classifier predicts for a given observation in a dataset, but also on the certainty that the correct class is predicted. In practice, different types of errors result in different outcomes in the real world. In the medical field, making a false negative prediction while testing for chronic kidney disease leads to a serious challenge. Consequently, making a wrong prediction on the likelihood of whether a patient is suffering from chronic kidney disease or not will lead to other health issues [1], particularly in low-income countries where millions die due to lack of proper medication and treatment.

Imbalanced class distribution is a common problem in chronic kidney disease classification, where chronic kidney disease patients or the negative class could greatly outnumber the chronic kidney disease patients or the positive class. Imbalanced class distribution has become one of the major challenges in machine learning; therefore improving the predictive accuracy of kidney disease classification models is an ongoing research problem [2]. The performance of standard machine learning algorithms tends to decrease when trained on a dataset that contains imbalanced class distributions. As a result, imbalanced class distributions particularly the frequent occurrence of negative class affects the classification performance of the positive class, which is critical for disease detection. Handling imbalanced datasets and improving the classification accuracy of imbalanced datasets needs additional research effort [1, 2]. Numerous methods have been proposed to improve the predictive accuracy of imbalanced dataset classification such as changing the class weight or class distribution [3].

The availability of a large number of medical datasets related to chronic kidney disease and the development of predictive models for chronic kidney disease detection with sampled data representing the overall population has become common practice in developing a predictive model for disease diagnosis. Although many research works have focused on chronic kidney disease prediction, comparing the predictive accuracy of classification models, data sample design, and the effect of imbalanced class distribution on the performance of predictive models has been neglected. Few studies have been conducted to address the issue of imbalanced class distribution on the performance of classification models for the detection of chronic kidney disease. Andrić et al. [4] investigated the effect of imbalanced class distribution on the performance of classification models. However, the authors did not conduct any experiments on any medical dataset, as the study focused on credit card assessment. Moreover, the problem of imbalanced class distribution was not empirically evaluated for chronic kidney disease detection. Furthermore, despite the importance and applicability of automated disease diagnosis, better resampling techniques were not used.

To address the perceived research gap, this paper examines the effect of imbalanced class distribution on the predictive accuracy of selected classification models. In addition, the effect of oversampling and sampling techniques on a real-world chronic kidney disease dataset was examined using gradient boosting, decision tree, random forest, and k-nearest neighbor (kNN). The performance of the proposed model is evaluated with uncertainty measures such as decision function and prediction probability of the positive and negative classes. The main contribution of this study is summarized as follows: (1) The study showed that the examined resampling techniques optimizes the prediction accuracy, (2) evaluation of gradient boosting, decision tree, random forest, and kNN was performed for the imbalanced class, and (3) investigation was performed using the minimum class distribution size for the imbalanced class distribution using the dataset.

These remaining sections of the paper are organized as follows: section 2 presents a review of related works, while the classification algorithms are discussed in section 3. The experimental framework is described in section 4, and the section also presents the experimental results, while section 5 presents the conclusion and recommendation for future work.

This section presents the method employed for data collection and analysis. In addition, the statistical method employed for analyzing the dataset was presented. Furthermore, the performance metrics employed in the evaluation of the performance of the employed machine learning algorithms are discussed.

3.1 Data collection

To conduct the experiments, real-world chronic kidney disease data was collected from Kaggle online data repository. The characteristics of the dataset employed in the study are summarized in Table 1. The dataset has 399 observations of which 149 observations are for chronic kidney disease patients and 250 observations are not for chronic kidney disease patients (non-patient observations). Each observation has 18 features which are used for describing the class label.

Table 1. Summary of data sample

The acquired dataset contains information on chronic kidney disease. Features extracted include demographic information of patient such as age and laboratory test results. The laboratory test result features include blood pressure, sugar level, albumin, blood glucose random test, potassium, sodium, blood urea, red blood cell count, and hemoglobin.

3.2 Descriptive statistics of numeric features

The descriptive statistics include the mean, median, maximum, minimum, and standard deviation for each input feature of the chronic kidney disease dataset. The descriptive statistics for each numeric input feature in the dataset are summarized in Table 2.

Table 2. Descriptive statistics for numerical input features

We observe from Table 2, that the maximum age value is 90 and the minimum age in the observation is 2. This is because, the dataset includes all of the age groups ranging from children to the elderly.

3.3 Feature selection

To reduce the dimensionality and remove irrelevant features that negatively affects the performance of the classification model, a sequential feature selection algorithm was employed. The sequential feature selection algorithm is widely used for selecting the optimal number of feature subsets that are relevant to train the model. The sequential feature selection algorithm removes or adds one feature at a time based on classifier performance. The number of features added is determined a priori; thus, the algorithm terminates when the pre-defined number of feature subsets to be selected is reached.

3.3.1 Optimal feature selection with sequential feature selection

A sequential feature selection algorithm is employed for finding the optimal set of input features from the 18 features characterizing each of the observations in the dataset. The sequential feature selection algorithm returned the feature subset described in Table 3 as the optimal feature set. Hence, the classification algorithms are trained on the optimal feature set summarized in Table 3. A classification accuracy of 99.3% is achieved with the sequential feature selection algorithm when the selected features are used for training as depicted in Figure 1. The best combination is given as: 0, 1, 2, 3, 4, 6, 8, 9, 10, 11, 13, 14, 15 achieving an accuracy of 99.3%.

Table 3. Optimal input features selected by the sequential feature selection algorithm

As presented in Table 3, the features such as age, blood pressure, and so on, are selected as the optimal feature set. The optimal features are used as input features for training the classification algorithms to achieve a better predictive performance. The features presented in Table 3, consists of only 13 features out of the total 18 features which characterize the dataset and are used when the classification algorithm is trained on the training set. As observed from Table 4, the classification accuracy of the model tends to improve with additional features. However, the highest accuracy was achieved with 13 features out of the 18 features. Figure 1 demonstrates the cross-validation score of the predictive model for chronic kidney disease detection.

1.png

Figure 1. Cross-validation score of different feature combination selected using sequential feature selection

It was observed that the maximum cross-validation accuracy is achieved with 13 input features for characterizing each observation of the dataset as depicted in Figure 1. Moreover, the cross-validation accuracy remained constant for feature subsets 14 to 18. Hence, the maximum number of input features that maximizes the predictive accuracy of the classification model is 13 and the highest cross-validation score is 99.3%. The best combination of the 13 input feature subset are as follows: Best combination: 0, 1, 2, 3, 4, 6, 8, 9, 10, 11, 13, 14, 15. This combination gives an accuracy of 99.3%.

3.4 Synthetic minority oversampling (SMOTE)

Training a standard algorithm on a dataset with imbalanced class distribution leads to performance problems of the minority class in the training set [13]. Therefore, the synthetic minority oversampling technique (SMOTE) is commonly applied for imbalanced classification problems to alleviate the effect of the class imbalance on the performance of the predictive model. The synthetic oversampling technique introduces new instances of the minority class to balance the distribution.

Real-world datasets such as telecommunication, fraud detection, and medical diagnosis datasets usually have a higher number of observations of a given class under-sampled compared to the other classes [14]. An imbalanced dataset substantially compromises the machine-learning algorithm since most machine-learning algorithms expect balanced class distribution or an equal miss classification cost. For this reason, as mentioned earlier, numerous approaches have been proposed to handle imbalanced datasets. In this study, we present a synthetic minority oversampling approach. The synthetic minority oversampling approach is mathematically described as follows.

$\mathrm{B}_{\mathrm{r}}=\left|\frac{\alpha}{\mathrm{y}}\right|$    (1)

where, Br is the balancing ratio given by Eq. (1), B represents the imbalanced dataset, alpha indicates the subset of the sample belonging to the minority class, and y indicates the subset of the sample belonging to the majority class. The balancing process is equivalent to resampling r to a new dataset rnew, where B_r> rnew.

3.5 Random majority under sampling

The majority under-sampling is the process of balancing the dataset by randomly reducing samples from the majority class. A number of practical classification problems contain imbalanced class distributions [14]. There are two resampling techniques to handle imbalanced datasets: oversampling and undersampling techniques. Oversampling increases the number of minority class samples until the class distribution is balanced. The most employed oversampling technique is the synthetic minority over-sampling technique (SMOTE). In contrast, majority oversampling reduces the samples from the majority class until the class distribution is balanced. The most common under-sampling technique is random sampling, where the samples of the majority class are randomly removed until the class distribution is balanced.

3.6 Evaluation criteria

The most important predictive performance evaluation metric for evaluating a classifier is the uncertainty estimate of predictions [15]. To test the uncertainty of the classifier we employed prediction probability as performance metrics. Other useful metric used for the evaluation of the model is the accuracy score and receiver operating characteristics curve (ROC). The accuracy and ROC curve of a classifier is defined by Eq. (2).

Predicitve Accuracy $=\frac{T}{N}$     (2)

where, T is the number of correct predictions and N is the total number of predictions (correct and incorrect) made by a given classification model. In an imbalanced dataset, the type of errors is more due to a more frequent number of one class than the other class. This is very common in practice specializing in medical datasets where the data points in the positive class are lower than the negative [16]. In such cases, accuracy might not sufficiently describe the performance score of a classification model. Accuracy is an inadequate measure of performance to quantify the predictive model performance on imbalanced chronic kidney disease classification [17]. Thus, the receiver operating characteristics (ROC) curve is employed for evaluating the classification performance of the model on chronic kidney disease detection. In addition, the confusion matrix is employed as an alternative performance measure to further investigate the model performance for imbalanced chronic kidney disease dataset classification. One of the most comprehensive ways to present the result of evaluating a binary classification model on an imbalanced classification task is a confusion matrix [18]. The confusion matrix is a two-by-two array where the rows correspond to the true class (ground truth) and the column corresponds to the predicted class.

The observations correctly predicted as chronic kidney disease patients are true positive (TP) and the observations correctly predicted as not chronic kidney patients are true negative (TN). In contrast, the observations incorrectly classified as chronic kidney disease patients are false positive (FP), and observations incorrectly predicted as not chronic kidney disease patients are true negative (TN). The TPR shows the fraction of true positives among true positive predictions given by Eq. (3).

$\mathrm{TPR}=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}}$   (3)

In contrast, the FPR shows the fraction of false positive among the false negative predictions. The FPR is given by Eq. (4).

$\mathrm{FPR}=\frac{\mathrm{FP}}{\mathrm{TP}+\mathrm{TN}}$      (4)

Precision: By precision defines the percentage of true positives, number of true positive (NTP), among all examples that the classifier has labeled as positive: NTP and number of false positive (NFP). The value is thus obtained by using the formula given in Eq. (5).

Precsion $=\frac{\text { NTP }}{\text { NTP }+\text { NFP }}$   (5)

In this study, a comprehensive experimental study on the effect of resampling and imbalanced class distribution of chronic kidney disease prediction was presented. Support vector machine (SVM), random forest, k-nearest neighbor (kNN), and adaptive boosting algorithms are applied to a real-world chronic kidney disease dataset obtained from Kaggle online data repository. In addition, examined how the input feature quality affects the models performance for classification.

The simulation result shows that the SVM, decision tree, random forest, and kNN models have better performance when trained on optimal features selected through a sequential feature selection algorithm. Furthermore, we have explored the effect of skewness of class distribution on the performance of standard machine learning models. Finally, we have implemented the state-of-the-art chronic kidney disease prediction model which achieved a classification accuracy of 99.3%. Furthermore, we have analyzed the models performance for risky medical decision-making with probability calibration. The results show that the model is robust and can be applied in medical decision support systems for the diagnosis of chronic kidney disease.

The limitation of this study is that the study was conducted only on a chronic kidney disease dataset. In addition, the study employed five supervised learning algorithms for comparison. In future work, the authors will test the resampling techniques with other supervised learning methods such as logistic regression, Naïve Bayes, and deep learning methods to investigate the effectiveness of the resampling technique.