Phishing Detection Using Random Forest-Based Weighted Bootstrap Sampling and LASSO+ Feature Selection

Phishing attacks have evolved significantly, making it increasingly difficult for users to differentiate between legitimate and malicious websites. According to the Anti-Phishing Working Group (APWG), the number of phishing incidents has increased by 15.87%, from 832,000 in 2022 to 964,000 in 2024 [1-3]. These attacks endanger sensitive data and erode users' trust in online platforms and services [4]. Phishing websites use a technique of URL concealment to trick users and bypass traditional security measures [5-7]. The changing nature of phishing URLs adds another complexity to their detection. Feature selection techniques can address this issue by reducing complexity and enhancing the model's capacity to accurately differentiate between various categories of websites [8]. For instance, Hannousse and Yahiouche [8] demonstrated that implementing feature selection strategies improved accuracy to 96.86%. Therefore, there is a need for more effective detection and protection systems.

Traditional feature selection methods often involve trade-offs between predictive performance and execution time. Achieving high accuracy frequently requires more features, which increases computational cost and training time [9]. Reducing features may improve computational efficiency, but it risks eliminating key characteristics and potentially lowering accuracy [8]. These limitations make traditional approaches inadequate for handling complex datasets where both high accuracy and computational efficiency are essential. To overcome these limitations, we introduce LASSO⁺, a feature selection method designed to optimize performance and efficiency simultaneously. In contrast to previous studies [10] emphasizing feature count, we focus on balancing performance and efficiency. This method ensures that computational resources are not overburdened while maintaining high accuracy, aligning with Pudjihartono et al., who emphasized balancing feature selection to optimize both performance and computational resources [11].

LASSO⁺ is an advanced feature selection method combining LASSO (Least Absolute Shrinkage and Selection Operator) with correlation thresholding and Grid Search to enhance feature selection efficiency. Details of its implementation are provided in Section 3.2. Correlation thresholding helps to eliminate highly correlated features, while Grid Search fine-tunes the regularization parameter (λ) to optimize feature selection. Balancing these trade-offs between recall, performance, and computational efficiency is critical, as highlighted by recent studies [11, 12]. This method balances performance and computational efficiency by automatically selecting the most important features during learning, reducing the impact of less important ones, and simplifying the model without losing effectiveness [11, 13-17].

In phishing detection, recall—also known as the True-Positive Rate (TPR)—is essential because it ensures that all potential threats are detected by minimizing the risk of missing any phishing attempts. The primary objective is to identify every possible threat [15]. Even though emphasizing recall might increase computational demands, it remains a critical priority. Missing a single phishing attempt may result in significant security threats.

Another issue in phishing detection is managing the high proportion of outliers in the datasets. More than 70% of the data points were identified as outliers using the Interquartile Range (IQR) method. This proportion of outliers can impact the performance of traditional machine learning models and often leads to inaccurate predictions [16, 17].

To solve these problems, the LASSO⁺ model integrates with Random Forest (RF) enhanced by Weighted Bootstrap Sampling (WBS) [18]. This integrated method improves predictive performance and effectively handles imbalanced and outlier-prone phishing datasets. WBS addresses the problem of outliers by ensuring a balanced sample representation during model training, which enhances recall and overall accuracy. Moreover, RF is well-known for its accuracy in multi-class datasets, user-friendliness with different dataset sizes, and stability. Conventional feature selection methods often necessitate a trade-off between maximizing predictive performance and minimizing execution time, which this study aims to address [10, 14, 19, 20].

The primary contributions of this study are threefold. First, we introduce and carefully evaluate the LASSO⁺ feature selection method for phishing detection. This method strategically combines LASSO with correlation thresholding and Grid Search to optimize feature selection. Second, we investigate the impact of integrating Weighted Bootstrap Sampling with Random Forest. We assess its effectiveness in improving model predictive performance, especially in handling outliers phishing datasets. Third, we provide a detailed analysis of the trade-offs between predictive performance and execution time. This analysis offers insights into how these factors can be effectively balanced to achieve superior performance in phishing detection systems.

The proposed method, as illustrated in Figure 1, is employed to enhance the performance of phishing website detection. This process includes data preparation, advanced feature selection, and robust classification techniques. It begins with data preparation, where duplicate entries and null values of the dataset are removed. During this stage, outliers are identified and permitted to be accounted for in subsequent analysis. Afterward, the data is normalized using a Min-Max Scaler. This scaler adjusts all features to the same scale between 0 and 1. It provides a simple feature range without introducing negative values or producing large ranges. Thus, the model is easier to interpret and reduces computational complexity.

Once the data is prepared, the study moves on to LASSO⁺ feature selection. The process starts with Pearson Correlation to identify and measure the linear relationship between features. Features with high correlations are subsequently subjected to Correlation Thresholding. In Correlation Thresholding, redundant features are removed to reduce multicollinearity and simplify the model. Grid Search is employed to optimize the parameters for LASSO and refine the feature selection further. This ensures that only the most relevant features are selected efficiently. Then, LASSO is employed to select the final set of features.

1.png

Figure 1. Proposed method for phishing detection using Random Forest based on LASSO⁺ feature selection

The selected features are then fed into the RF model, which is enhanced by WBS based on Uncertainty. This technique prioritizes samples that are difficult to classify, with one significant factor being outliers identified through the data preparation stage. Within the RF model, subset feature selection is applied at each node of the decision trees. Rather than using all available features, the model randomly selects a subset of features at each node to reduce overfitting and increase model diversity.

WBS normalization adjusts each sample's weights based on uncertainty to ensure their sum equals one. Then, the decision trees were built using these weighted samples and the selected subset of features. Each tree is constructed by splitting nodes based on a subset of features with higher uncertainty values. The final prediction is determined by aggregating the predictions from all decision trees in the forest using majority voting, where the class predicted by most trees is selected as the outcome.

Finally, the model's performance is evaluated using a comprehensive set of metrics, including accuracy, precision, recall, F1-score, Area Under the Curve (AUC), and execution time. These metrics provide a holistic view of the model’s effectiveness. An ablation study is also conducted to compare the traditional LASSO method with the proposed LASSO⁺ method. This highlights the improvements in recall and computational efficiency. The methodology concludes with a Comparison of Performance across four datasets (A, B, C, and D).

3.1 Data collection

This study utilized several public datasets to evaluate the performance of different feature selection methods for phishing website detection. The datasets used are from Vrbančič (Dataset A) [33], Hannousse and Yahiouche (Dataset B) [8], Prasad and Chandra (Dataset C) [34], and Mohammad et al. (Dataset D) [35]. These datasets are widely used in phishing detection research. For example, Dataset A has been utilized in various studies as a benchmark for evaluating feature selection methods in phishing detection [31, 36, 37]. Dataset B is often referenced in studies analyzing phishing characteristics [10, 38-40]. Due to its large size and inclusion of modern attributes such as obfuscation and media elements, Dataset C has been used to test advanced machine-learning algorithms for phishing detection [41]. Finally, Dataset D represents a smaller but balanced dataset [3, 42]. The details of these datasets are summarized in Table 1.

Table 1. Dataset information

The datasets vary in feature count, class distribution, and overall size, reflecting diverse characteristics essential for evaluating feature selection methods. Dataset A has the highest feature count and a noticeable imbalance, with legitimate samples dominating. In contrast, Dataset B provides a balanced class distribution. Dataset C is the largest in size, exhibits moderate imbalance, and incorporates modern phishing characteristics. Dataset D, with the smallest feature count and near-balanced classes, offers efficiency for lightweight evaluations. These differences highlight the complementary nature of the datasets, ensuring robust and comprehensive testing under varied conditions.

3.2 Data preparation

This study focused on handling outliers in the datasets. Outliers may impact the performance of ML models. More than 70% of the data were identified as outliers using the IQR method. Figure 2 presents the outlier sensitivity graph for each dataset, illustrating the distribution of inliers and outliers. In the graph, inliers are represented by yellow bars, while outliers are shown in orange bars. It underscores the critical issue of outliers and their substantial impact on model performance, which this study aims to address. Therefore, this study strongly aims to address outliers and their impact on model performance.

2.png

Figure 2. Outlier sensitivity for across the dataset

Data preparation in this study involves several steps. First, duplicate entries are removed to prevent distortion caused by overrepresented data points. Next, missing values are addressed to minimize bias and maintain the model's accuracy. Once the data is cleaned, feature normalization is applied using the Min-Max Scaler to rescale all feature values into a consistent range, typically between 0 and 1 [43]. The formula for Min-Max normalization is as follows: Eq. (1).

$X_{\text {norm }}=\frac{X-X_{\min }}{X_{\max }-X_{\min }}$                 (1)

where, X is the original feature value, and X_min and X_max are the minimum and maximum values of the feature, respectively. This formula ensures that all feature values are scaled proportionally within the defined range and prevents features with larger numeric ranges from dominating the learning process. For example, if the URL length has a minimum value of 11 a maximum of 57, and an observed value of 48, the normalized value would be:

$X_{\text {norm }}=\frac{48-11}{57-11} \approx 0.804$               (2)

Normalization scales all feature values to a non-negative range and standardizes their scale without removing outliers. This ensures equal contribution from small-range features like URL length and large-range features like timestamps to the learning process [43]. This process enhances the model's stability and ensures fairness in feature contribution during learning.

3.3 Feature selection and optimization process

The feature selection and optimization process begin with Correlation Thresholding to identify and remove redundant features. This step reduces multicollinearity by analyzing the linear relationships between features and eliminating those with high correlation. Next, Grid Search optimizes the regularization parameter λ in the LASSO model. LASSO applies this optimal λ to shrink less important feature coefficients to zero, retaining only the most relevant features. This process improves both model performance and computational efficiency.

3.3.1 Pearson correlation

The feature selection process begins with Pearson correlation, which quantifies the linear relationship between pairs of features in the dataset. The Pearson correlation coefficient $r_{i j}$ ranges from -1 to 1 , where $r_{i j}=1$ indicates a perfect positive relationship, $r_{i j}=-1$ indicates a perfect negative relationship, and $r_{i j}=0$ indicates no relationship. The coefficient is calculated using the Eq. (3) [44].

$r_{i j}=\frac{\sum_{k=1}^n\left(X_{i k}-\bar{X}_i\right)\left(X_{j k}-\bar{X}_j\right)}{\sqrt{\sum_{k=1}^n\left(X_{i k}-\bar{X}_i\right)^2} \sqrt{\sum_{k=1}^n\left(x_{j k}-\bar{X}_j\right)^2}}$                  (3)

where, $X_{i k}$ and $X_{j k}$ are the values of features $X_i$ and $X_j$ for the $k$-th observation, $\bar{X}_i$ and $\bar{X}_j$ are their mean values, and $n$ is the number of observations.

A correlation matrix is computed for all feature pairs to identify and remove redundant features. Threshold values $(\tau)$ are tested to find the optimal value that maximizes model performance. Table 2 outlines detailed steps for this process.

3.3.2 Correlation thresholding

Correlation Thresholding is applied after calculating the correlation matrix to identify and remove highly correlated features. Instead of using a fixed threshold, such as 0.9 , this study optimizes the $\tau$ based on model performance. Threshold values ranging from 0.5 to 1.0 are tested to find the optimal $\tau$. For feature pairs with a correlation $\left(\left|r_{i j}\right|>\tau\right)$, one feature is removed.

The decision on which feature to remove is based on three factors [45]. First, domain knowledge ensures the retention of features relevant to phishing detection. Second, feature importance scores, derived from preliminary analyses, prioritize features with higher contributions to model performance. Third, multicollinearity is reduced by carefully selecting and removing highly correlated features. Detailed steps for this process are outlined in Table 2.

Table 2. Pseudocode for correlation thresholding

3.3.3 Grid search for lasso optimization

Grid Search is used to optimize the $\lambda$ in the LASSO model. The parameter $\lambda$ determines the strength of the penalty applied to feature coefficients, where larger values shrink more coefficients to zero. This simplifies the model by removing less important features while retaining the most relevant ones [46].

Table 3. Pseudocode for grid search

To find the optimal $\lambda$, Grid Search tests a predefined range of values. Each $\lambda$ is evaluated using cross-validation to measure its impact on model performance. The value that achieves the best balance between accuracy and simplicity is selected as the optimal $\lambda$. The detailed steps of this process are outlined in Table 3.

3.3.4 LASSO feature selection

After determining the optimal $\lambda$ through Grid Search, LASSO applies this parameter to penalize less important features in the model. The LASSO objective function is defined in Eq. (4) [46],

$\hat{\beta}_{\text {lasso }}=\operatorname{argmin}_\beta\left\{\frac{1}{2 N} \sum_{i=1}^N\left(Y_i-\beta_0-\sum_{j=1}^p r_{i j} \beta_j\right)^2+\lambda \sum_{j=1}^p\left|\beta_j\right|\right\}$            (4)

This objective function minimizes the sum of squared errors and applies an L₁ penalty proportional to the absolute values of the coefficients $\beta_j$. The penalty term, controlled by $\lambda$, ensures that coefficients for less important features are shrunk to zero, effectively removing them from the model. The terms are defined as follows:

The first term represents the least squares error. This term calculates the sum of squared differences between observed and predicted outcomes, see Eq. (5).

$\frac{1}{2 N} \sum_{i=1}^N\left(Y_i-\beta_0-\sum_{j=1}^p r_{i j} \beta_j\right)^2$               (5)

where, $N$ is the number of observations, $X_{i j}$ is the value of the $j$-th feature for the $i$-th observation, $\beta_j$ is the coefficient associated with the $j$-th feature, and $\beta_0$ is the intercept term.

The second term is the L₁ regularization term. This term adds a penalty proportional to the sum of the absolute values of the coefficients $\beta_j$, see Eq. (6).

$\lambda \sum_{j=1}^p\left|\beta_j\right|$                (6)

The regularization parameter $\lambda$ controls the strength of this penalty. A larger $\lambda$ increases the penalty, shrinking more coefficients to zero and removing the corresponding features from the model. Conversely, a smaller $\lambda$ retains more features with non-zero coefficients.

The steps for applying for LASSO in feature selection are outlined in Table 4.

Table 4. Pseudocode for LASSO feature selection

Optimizing LASSO can be computationally expensive due to iterative Grid Search. This study simplifies the process to reduce execution time while maintaining model accuracy. The optimization ensures efficient and effective feature selection without sacrificing performance.

3.4 Random Forest

This study uses RF combined with WBS based on uncertainty to improve the performance and efficiency of phishing detection in complex datasets. It starts with LASSO⁺ feature selection, which identifies the most relevant features for classification. Then, these selected features are used in the RF model. These features are further evaluated and chosen during the building of decision trees.

3.5 Random Forest construction using Weighted Bootstrap Sampling with uncertainty

During the construction of each decision tree T_b in the forest, the dataset is resampled using WBS based on uncertainty sampling approach discussed by Liu and Li [18]. Instead of standard Bootstrap Sampling, WBS assigns a weight W_i to each sample based on the uncertainty of its predicted outcome. Those with higher uncertainty receive greater weights, so that these samples could be selected during the resampling process.

The weight W_i for each sample is calculated using the following formula:

$W_i=\frac{1}{\left|P\left(\hat{Y}_i=1 \mid X_i\right)-0.5\right|+\epsilon}$                  (7)

where, $P\left(\hat{Y}_i=1 \mid X_i\right)$ is the predicted probability that the sample $X_i$ belongs to the positive class, such as predicting that a website is phishing. The expression $\left|P\left(\hat{Y}_i=1 \mid X_i\right)-0.5\right|$ calculates the absolute difference between the predicted probability and 0.5 . This difference shows how unsure the model is about the prediction. If the value is close to 0.5 , it means the model is very uncertain. The small constant $\epsilon$ is added to avoid dividing by zero.

Once the weights are calculated for all samples, these weights are normalized so that the sum of all weights equals one. This normalization step ensures that the weights can be interpreted as probabilities when selecting samples. The probability of selecting a sample $X_i$ to be included in the new bootstrap sample $D_b$ is proportional to its weight $W_i$. This probability is given by:

$P\left(X_i \in D_b\right)=W_i$            (8)

Using these weights, the new bootstrap sample $D_b$ is created by randomly selecting $N$ observations from the dataset. The probability of each observation being selected depends on its weight $W_i$. Hence, samples with higher uncertainty are more likely to be included in the training set. This approach helps the model to focus on more challenging cases, ultimately improving its overall performance.

In RF model, each decision tree is constructed by selecting a random subset of features at each node. Instead of using all available features in the dataset, the model selects a smaller random subset of features to consider when splitting at each node. The number of features selected at each node is denoted by m, while M represents the total number of features available in the dataset. The model then examines the selected features m to find the one that best divides the data at that node. To determine how many features m should be selected at each node, the commonly used formula:

$m=\sqrt{M}$            (9)  

The final prediction is determined by combining all these predictions after each decision tree in the forest makes its prediction for a sample $X$. The following explanation shows how the sample $X$ is being predicted. The prediction made by the $b$-th tree is denoted as $T_b(X)$. The final predicted class for the sample is represented by $\hat{Y}$.

In a classification model, these predictions are combined using majority voting. This means the class often predicted by the trees becomes the final prediction. The equation is:

$\hat{Y}=\operatorname{mode}\left\{T_h(X)\right\}_{h=1}^B$            (10)

In Eq. (10), B stands for the total number of trees in the forest. The mode function selects the class that appears most frequently among the predictions from all B trees. The RF formula is widely recognized and applied in [47].

3.6 Model training and evaluation

After resampling the data using WBS, the RF model is trained on the newly constructed dataset to enhance its ability to detect phishing websites in imbalanced datasets. The model's performance is evaluated using accuracy, precision, recall, f1 score, AUC, and execution time.