Random Forest and One-vs-Rest Classification for Malware and Ransomware Detection

Cybersecurity has been at a critical stage where the quantity, sophistication, and occurrence of malware and ransom attacks have grown to proportions that cannot be dealt with using conventional defense tools. Modern threat actors are increasingly using polymorphic code, file-less attacks, and artificial intelligence (AI)-based avoidance techniques, which take advantage of weak points in traditional signature-based scanners and heuristic rules, increasing the risk facing the organization. These dangers are reflected in mass data breaches, financial damages, compromised services, and cloud and Internet of Things (IoT) systems [1, 2]. The fast growth in the number of interconnected devices, fast networks, and automated systems increases the attack surface, making manual threat detection ineffective and requiring intelligent and automated defense models. Traditional signature-based antivirus solutions are only efficient in countering the established threat types but essentially useless in identifying zero-day malware versions and fast-evolving ransomware families [3]. Some of these limitations are overcome by heuristic detection methods by analysing the behaviour patterns, although they often have high false-positive rates, are computationally intensive, and are not adaptable to changing threat scenarios. Machine learning (ML) is also a promising technology in malware detection due to its ability to learn discriminative features, detect anomalies, and generalise over unknown types of attacks. However, the challenge of critical class imbalances, adversarial images, redundant features, and low multi-class classification rates with different cyber threats continues to face malware classification with ML, even today [4, 5].

These shortcomings need to be overcome with effective malware detection systems that combine adaptive learning, balanced data representation, optimised features, and scalable classification architecture. The current paper advances a multi-class detection system based on a Random Forest Classifier (RFC) and a One-vs-Rest (OvR) multi-class testing approach to improve the detection accuracy, interpretability, and resiliency. The combination of Synthetic Minority Oversampling Technique (SMOTE)-based data balancing, features extraction using the Random Forest (RF) importance measures and Principal Component Analysis (PCA), and multi-class discrimination using OvR will make the system more reliable in detecting the heterogeneous threats such as cross-site scripting, Distributed Denial of Service (DDoS) and Denial of Service (DoS), password attacks, phishing, Structured Query Language (SQL) injections, zero-day exploits, and ransomware. As opposed to the previous ones that utilize either a single-model classification or unbalanced datasets, the given approach is based on an end-to-end pipeline that includes the preprocessing of data, adaptive learning, and a framework that can be extended to incorporate real-time threat intelligence updates, along with comprehensive performance assessment. The system has a much higher accuracy and strength through precision, recall, F1-score, Area Under Curve of the Receiver Operating Characteristic (AUC-ROC) analysis, and confusion matrix analysis. This framework achieves the combination of ensemble learning with multi-class classification and balanced data representation to mitigate a major set of constraints within current techniques and provide an efficient, lightweight, and scalable endpoint security.

1.1 Objectives

The main purpose of this investigation involves developing an AI-controlled malware detection framework that enhances its ability to detect malicious signals while eliminating incorrect false alarms. The main purpose of the given research is to create an ML-based malware and ransomware detection system that enhances the detection accuracy without increasing false alarms. The following are the specific objectives of research:

• To develop a working malware detection model by using the RF model to classify the benign and malicious activities with the help of network, protocol, and behavioral features.
• To improve the multi-class attack detection through the OvR classification approach, allowing it to discriminate malware and ransomware of various categories.
• To enhance the performance and strength of classification through SMOTE-based data balancing and feature selection through RF feature importance and PCA.
• To minimize false positives and false negatives by making use of ensemble learning and optimal preprocessing methods that are applicable in endpoint security settings.
• To enhance the interpretability of the models by assessing the performance based on the confusion matrices, ROC curves, and conventional measures, i.e., accuracy, precision, recall, and F1-score.

1.2 Structure of the paper

In this paper, it has been noted in the introduction that the challenges presented by modern malware and ransomware attacks were increasing, thus explaining the necessity of a flexible detection framework. The literature review is a survey of extant signature-based, behavioral, and machine-learning strategies, revealing salient knowledge gaps and the need to thoroughly evaluate them. One section describes the methodology of the proposed system that includes the data preprocessing, the process of class balancing using SMOTE, the feature selection algorithms, and the process of working with RF and OvR classifiers. The results and discussion section provides empirical results, which are used to measure the results of a model using accuracy, precision, recall, F1-score, ROC curves, and confusion matrices. Lastly, it concludes the work by presenting the input of the given research and giving potential future research directions, such as the incorporation of deep learning methods and the development of adversarial robustness.

A malware detection framework uses a consistent methodology that combines data preprocessing steps with feature selection, along with ML classification, together with real-time threat intelligence updates to boost cybersecurity defenses [21]. The framework works at an enterprise level for malware detection and mitigation because it delivers high accuracy together with adaptability and scalability. Two parts comprise the methodology explanation, demonstrated sequentially throughout this section.

Figure 1. Block diagram

The initial part of malware detection processing uses data preprocessing to ready raw data for ML models through handling missing values, combined with encoding categorical attributes and scaling numerical dimensions and class balancing techniques [9]. The dataset D contains n samples that consist of m feature variables structured as x_i∈Rm with corresponding binary labels y_i∈{0,1}, indicating malware-classified instances and non-malware instances. Statistical techniques, including mean and median imputation, help replace missing data values in order to maintain complete participation in training procedures, as shown in Figure 1. The ML model requires binary vectors as input from one-hot encoding, which transforms categorical data for interpretation purposes. The z-score normalization computes standardized features through the following formula:

$\mathrm{X}_{\text {Scaled }}=\frac{\mathrm{X}-\mu}{\sigma}$            (1)

Both μ signifies the mean and σ stands for standard deviation in the formula. The standardization technique keeps all features balanced so the learning process avoids scale-based bias. The main problem in malware class identification rests in the huge disproportion between benign files and malware files, causing predictions to become disproportionately biased [22]. The SMOTE serves as the solution to balance class distribution. SMOTE creates artificial samples of the minority class malware by computing the midpoint between real-class samples. A synthetic sample gets produced using the malware example x_minor along with the nearest neighbor x_neighbor.

$x_{\text {new}}=x_{\text {minor}}+\lambda\left(x_{\text {neighbor}}-x_{\text {minor}}\right)$             (2)

A value λ∼U(0,1) was randomly sampled from the uniform distribution. The method guarantees malware samples have optimal representation within the dataset, which leads to better model generalization.

A set of feature selection methods reduces unimportant or duplicate features that keep vital indicators for malware detection after dataset preprocessing [23]. The malware detection framework implements an RFC as the classifier while using OvR for multi-class malware classification enhancement.

3.1 Reproducibility details: Dataset, features, and evaluation protocol

A publicly available network intrusion dataset made available to academic research was experimented with, and a curated subset of about 150 labeled data in nine attack types and benign traffic was revealed to researchers. There was an imbalance in the classes of the original data, which was resolved by applying SMOTE only to the training folds, making the resulting data nearly balanced in terms of classes after over-sampling. It has features such as network identifiers (encoded sender/receiver identities and IP representations), protocol and transport features (protocol type, ports, packet size), and behavioral features such as flow statistics and flag features; categorical variables were one-hot encoded, and numerical features were z-score normalized. RF importance was used to rank feature relevance, and PCA was then applied to reduce dimensions. To assess the models, stratified k-fold cross-validation (5-fold in RF and 10-fold in OvR) with fixed random seeds was used to ensure that the results were reproducible, and the results of the evaluation were reported in terms of accuracy, precision, recall, F1-score, AUC-ROC, and confusion matrices.

3.2 Random Forest Classifier

RFC serves as an ensemble learning model for malware detection to build stronger accuracy and robustness in the classification outcomes. Researchers start the process by selecting features from the dataset, which includes properties of system behavior alongside network activity and file characteristics. Multiple decision trees (DT) receive their features through random distribution before undergoing training through distinct portions of the data, as shown in Figure 2.

Each DT learns unique patterns during its independent operation, which creates diverse proposed classifications. The aggregate voting decision of all DT determines the classification through the number of times one outcome is predicted by the majority. The model uses this procedure to eliminate biases across its trees and minimize overfitting [24]. The classification layer uses the group decision of the trees to determine whether an input sample is malware or benign. RFC uses a step-by-step procedure to improve its malware detection rates and function effectively as a dependable cybersecurity solution. The operation and deployment of the model achieve increased resistance to false positives while addressing new cyber threats through ensemble learning and majority voting techniques.

Figure 2. Random Forest Classifier (RFC) architecture for malware detection

RFC represents an ensemble learning technique that produces more accurate malware classifications through several DT aggregations. RF addresses information overfitting and high variance through multiple DT models, which use distinct subsets of data to create collective forecasting. RFC proves highly efficient for dealing with the typical cases of malware detection, which often include imbalanced data along with multivariable features as well as non-linear relationships [25]. RF's main benefit stems from its universal applicability across malware types, which happens through bootstrapping and feature randomness, while majority voting makes the classification model dependable and strong. RFC includes an organized set of procedures that perform classification work. Bootstrap sampling (bagging) generates numerous random subsets of data from the original dataset by using sampling with replacement as the first step. The independent DT receives individual subsets to train while maintaining different learning perspectives. RF differs from standard DT because it chooses random subgroupings of features during its node distributions to prevent tree correlation and boost predictive capability [26]. RF feature importance provides a ranking system that evaluates features based on their impact on model classification. The Gini impurity criterion applies to measure feature importance through the following calculation:

$G i n i=1-\sum_{k=1}^K P_k^2$            (3)

Each node contains a probability value pk that represents the probability of class k within the node. The decreasing effect of Gini impurity when splitting by a specific feature X_j is calculated through:

$\Delta G i n i\left(x_j\right)=$ Gini$_{\text {parent}}-\sum_{i=1}^{\text {childern }} \frac{N_i}{N} G i n i$             (4)

3.3 One-vs-Rest classifier

Multiple binary classifiers operate independently under the OvR classification framework to solve multi-class classification issues. Every classifier in OvR learns to distinguish exactly one class type while ignoring all other possible classes present in the dataset. This approach provides effective results for instances where users apply binary classification models such as logistic regression and support vector machine (SVM) or RF when handling multi-class problems [27]. The system starts with accepting feature vectors that include extracted attributes from the dataset. Each classifier receives different features from the K input classifiers to determine whether the input data belongs to its own class or not. The output of every classifier contains a probability value that shows the degree to which the analyzed data belongs in that particular class category, as shown in Figure 3.

Figure 3. One-vs-Rest (OvR) classification architecture for multi-class prediction

OvR classification represents a standard solution when handling multi-class problems through the creation of separate binary tasks for each class. The dataset contains N training samples made up of feature vectors alongside their corresponding class labels, and it takes this form:

$D=\left\{\left(x_1, y_1\right),\left(x_2, y_2\right), \ldots,\left(x_N, y_N\right)\right\}$            (5)

The model contains features x_i∈Rd while y_i∈{1,2,...,K} serves as the class label for determining the sample category. OvR classification trains individual binary classifiers that discriminate between each class and the entirety of the remaining classes. A binary classification task is built for each class k that uses y_k as its target variable [28]. A sample x_i with d dimensions enters as the input vector, which matches with the output label y_i among the set of categories K. Training K independent binary classifiers is the main purpose of OvR classification because each classifier needs to differentiate between one class and all remaining classes. The binary classification problem for every k uses y_k as the target variable, defined as:

$Y_k=\left\{\begin{array}{cc}1, & \text { if the sample belongs to class } k \\ 0, & \text { otherwise }\end{array}\right.$            (6)

3.4 Evaluation metrics

Accuracy: Accuracy is a metric that quantifies the degree to which a model's predictions are right. The computation involves dividing the sum of true positive (TP) and true negative (TN) forecasts by the total number of predictions produced, which include TP, TN, false positives (FP), and false negatives (FN). Mathematically, precision is precisely defined as:

Accuracy $=\frac{\mathrm{TP}+\mathrm{TN}}{\mathrm{TP}+\mathrm{TN}+\mathrm{FP}+\mathrm{FN}}$            (7)

Precision: Precision is a measure that calculates the ratio of correctly recognized positive predictions (TP) to all predictions classified as positive by the model. It demonstrates the precision of the model's favourable predictions [29]. Precision, in mathematical terms, is determined by dividing the number of TP by the total of TP and FP. The formula for precision is defined as:

Precision $=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FP}}$           (8)

Recall: Recall, often referred to as sensitivity or TP rate, is a quantitative measure that calculates the ratio of correctly predicted positive instances (TP) to the total number of actual positive cases. It denotes the model's capacity to accurately recognize all pertinent occurrences in a specific category. Mathematically, recall is computed by dividing the number of TP by the total of TP and FN. The recall formula is expressed as:

Recall $=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}}$             (9)

F1-score: The F1-score is a quantitative measure that calculates the harmonic mean of accuracy and recall, offering a balanced assessment of both. It is especially beneficial in situations when there is an imbalanced class distribution or where both FP and FN have equal significance. The F1-score may be expressed mathematically as follows:

F1-Score $=2 \cdot \frac{\text { Precision } \cdot \text { Recall }}{\text { Precision }+ \text { Recall }}$           (10)