A Risk-Aware and Explainable Machine Learning Framework for Phishing Website Detection Using Structural Website Features

The problem of phishing websites has been a key field of interest for cybersecurity research, as it continues to affect vulnerable users who enter bogus websites, leading them to share their personal information. With phishing techniques becoming more sophisticated, research has moved away from signature-based phishing identification and toward sophisticated, automated algorithms capable of recognizing phishing websites based on features of the URL, phishing website structure, data associated with the domain, and learning-based models [17, 18]. There are three types of current research studies: classical blacklist and heuristic methods, feature-based machine learning procedures, and more recent deep learning techniques [19]. This is a step towards a more dynamic detection model that is based on data, as opposed to static detection systems, and is better equipped to adapt to an evolving phishing threat.

2.1 Traditional phishing detection methods

Early phishing detection systems depended mostly on blacklist and heuristic-based techniques. Blacklist-based techniques detect phishing websites by comparing URLs with databases of previously identified malicious domains or webpages that are managed by security services, browser vendors, and other associated security platforms [20, 21]. These computationally efficient techniques are still commonly utilized in real security products and current web browsers. Nevertheless, their success is dependent on past reporting; therefore, they are often unreliable against recently generated phishing websites that have not yet been recognized and added to blacklist databases. To address these issues, instead of depending on existing recognized malicious content, heuristic-based solutions were developed to resolve some of the limitations by searching for suspicious URL patterns and webpage architecture. The general markers of suspicious website activities are strange, uncommon domain names, fake hyperlinks, and anomalies in URL patterns [22]. The efficacy of blacklist techniques is inferior to that of heuristic methods in identifying previously undetected phishing pages. Nevertheless, they are primarily dependent on manually constructed algorithms and constructed signals that require periodic updates due to the fact that phishing strategies are swiftly evolving [23]. This scenario complicates maintenance and may reduce the effectiveness of detection.

2.2 Machine learning approaches for phishing detection

Machine learning is used to detect phishing, so as to resolve the limitations of the blacklist and heuristic approaches. Thus, machine-learning-based systems’ classifiers are trained on labelled datasets comprising both phishing and genuine websites. They are able to discriminate phishing websites from legitimate websites by using URLs and domain characteristics [24]. Hence, machine learning is one of the crucial areas of study in the identification of phishing websites. Accordingly, an extensive variety of machine learning methods were examined. For the detection of phishing websites, Ramana et al. have introduced a collection of feature-selection approaches and reported higher accuracy in classification compared to baseline algorithmic methods [25]. Other research investigations have utilized extracted website features, like lexical, structural, and visual-layout characteristics. Additional methods include feature optimization methods that were employed to determine the optimal informative characteristics [26]. Moreover, comparative research indicates that ensemble learning enables significant performance benefits. A study was conducted to compare the results of a Voting Classifier with the results of the single classifiers, namely Decision Tree, Random Forest, Gradient Boosting, XGBoost, AdaBoost and Multilayer Perceptron. It has also simultaneously lowered the false positive and false negative results [27]. Random Forest in particular has been very popular due to its good prediction performance, robustness, and little overfitting [28]. The integration of feature selection methods with machine-learning classifiers is employed by hybrid methodologies to minimize feature redundancy while increasing classification accuracy [29]. The literature shows that despite the advancement of the artificial intelligence field, conventional and ensemble machine learning methods remain highly effective methods for the identification of phishing, especially when they are accompanied by the use of suitable feature engineering, feature selection, and controlled comparison of the models.

2.3 Feature-based phishing detection

Feature engineering is a vital aspect of phishing website identification, particularly in conventional machine-learning techniques. Numerous research studies divide phishing identification characteristics into three categories: lexical, host-driven, and content-driven, as retrieved from URLs and webpage architecture [30]. Lexical characteristics often define a URL's arrangement, such as its length, token structure, application of special characters, questionable prefixes or suffixes, and the use of an IP-address-based routing instead of a domain name in the URL. These characteristics are frequently relevant since phishing URLs often produce structural abnormalities designed to impersonate genuine domains while concealing harmful intentions. Host-driven characteristics offer extra data regarding the validity of websites. Domain registration characteristics, DNS-associated characteristics, SSL-related data, and other domain-level indicators that aid in the detection of questionable or temporary websites are examples of such qualities [31]. These markers have enhanced the identification of phishing attacks due to malicious operations. These indicators are useful for identifying newly registered or suspicious domains. Mohammad and McCluskey [32] created a phishing website dataset that is used widely in phishing detection studies. The assessment of machine-learning models for identifying phishing uses the University of California, Irvine (UCI) dataset that comprises 11,055 cases and 30 URL- and website-associated characteristics. Some studies have shown that structural and domain-related properties are significant indicators for phishing website detection. The continuous dependency on the sets of structured characteristics highlights the practical importance of defining the characteristics that most importantly affect the phishing website categorization.

2.4 Deep learning approaches

During the last few years, phishing detection researchers have been increasingly exploring deep learning algorithms to reduce the dependence on manual feature engineering and improve phishing prediction accuracy. Feature representations were automatically extracted from URLs and web content using deep neural networks, such as convolutional neural networks (CNNs) and recurrent neural networks [33, 34]. These methods are aimed at identifying patterns directly from the raw or somewhat processed input instead of relying entirely on the features that are engineered. CNN methods are often used with character sequences to learn phishing-related patterns without the need for explicit manual feature extraction from URLs. Other deep learning studies have used neural networks to extract URL attributes and then match them to the HTML or web page content to identify phishing-related structures and patterns on web pages [35]. These methods have yielded promising results and shown the potential of representation learning in detecting phishing attacks. Notwithstanding these benefits, deep learning techniques frequently need larger training datasets, extended training durations, and larger computational capacity than conventional machine-learning techniques [36, 37]. Thus, the feature-based machine learning models remain relevant in practical applications where interpretability, computational efficiency, and ease of implementation are important. Although significant research advances in deep learning have been made, traditional machine-learning methods are still an object of research interest.

2.5 Positioning of the present work

There are some limitations that are still present in the available literature. A lot of recent work has been done on complex deep learning architectures and hybrid or single machine learning classifiers. While these methods have shown some encouraging results, there is not much attention paid to indicators that are specific for phishing, such as false negative rates, in the majority of studies. For phishing website detection, this is a crucial concern, since a fraudulent phishing site might be marked as a genuine site and expose the user to a serious cyber threat. Moreover, it is difficult to compare previous studies because of variations in datasets, feature representations, data preprocessing and evaluation protocols. Thus, comparative evaluations under controlled conditions with the same data and identical experimental conditions continue to be valuable in both research and practical application. Furthermore, explainability and feature reduction remain relevant as phishing detection systems need to be practical, not only in terms of their performance but also in terms of having understandable indicators and avoiding unnecessary feature complexity. The current study is conducted for the purpose of addressing the aforementioned issues and compares the performance of several supervised machine learning classifiers such as Random Forest, SVM, Logistic Regression, Decision Tree, K-Nearest Neighbors, Naive Bayes and XGBoost in detecting phishing websites with a publicly available benchmark dataset of structural website features. The risk-aware false negative analysis, stratified cross-validation, compact feature subset evaluation, threshold optimization and permutation-based explainability aren’t only included because they’re simple, but because they align with the specific risk-aware requirements of this study. The goal of the current study, therefore, is to offer a more practical and understandable analysis of the machine learning models for detecting phishing websites.

This section discusses the experimental results obtained from phishing website detection by using a set of machine learning classifiers. The evaluation was carried out on the basis of conventional and risk-aware performance indicators. Besides the overall classification performance, particular focus was placed on the phishing-class recall and phishing FNR as it is a serious security risk to misclassify a phishing website as a legitimate one. This section includes model comparison, cross-validation analysis, compact subset of features evaluation, threshold optimization, and the analysis of explainability.

4.1 Risk-aware model performance comparison

The seven candidate classifiers were fitted using the 60% training partition and compared on the separate 20% validation partition under identical experimental conditions. Their performance was evaluated using accuracy, macro-F1, phishing recall, phishing FNR, and ROC-AUC. Macro-F1 served as the primary model-selection metric, while phishing recall and the phishing FNR provided security-oriented measures of missed phishing websites. Table 1 presents the validation performance of the evaluated classifiers. The final test partition was not used during this comparison or the subsequent configuration-selection procedures.

Table 1. Validation-set risk-aware performance comparison of classification models

Random Forest produced the highest observed validation accuracy of 97.24% and macro-F1 of 97.20%, together with a phishing recall of 96.02% and a phishing FNR of 3.98%. XGBoost and Decision Tree achieved the next-highest macro-F1 values of 95.77% and 95.69%, respectively. Although Naive Bayes achieved the highest phishing recall of 99.80% and the lowest phishing FNR of 0.20%, its substantially lower accuracy of 59.43% and macro-F1 of 55.72% indicate that this sensitivity was obtained at the cost of frequent false alarms involving legitimate websites. Based on the prespecified primary criterion of validation macro-F1, Random Forest was selected for the subsequent feature-subset, threshold, and explainability analyses. These values represent validation-set observations, while the selected configuration was evaluated separately on the untouched final test partition. Because XGBoost achieved the second-highest observed validation macro-F1, it was included in the paired statistical comparison. Random Forest correctly classified 51 instances that XGBoost misclassified, whereas XGBoost correctly classified 20 instances that Random Forest misclassified. The two-sided exact McNemar test was significant (p = 0.000303). Random Forest improved accuracy by 1.40 percentage points, with a paired-bootstrap 95% confidence interval from 0.68 to 2.17 percentage points, and improved macro-F1 by 1.43 percentage points, with a 95% confidence interval from 0.69 to 2.20 percentage points. Thus, the observed validation-set advantage of Random Forest over XGBoost was statistically supported. The paired validation comparison provided statistical support for the observed Random Forest advantage over Decision Tree. Random Forest correctly classified 48 instances that Decision Tree misclassified, whereas Decision Tree correctly classified 15 instances that Random Forest misclassified. The two-sided exact McNemar test was significant (p = 0.000038). Random Forest improved accuracy by 1.49 percentage points, with a paired-bootstrap 95% confidence interval of 0.81 to 2.22 percentage points, and improved macro-F1 by 1.51 percentage points, with a 95% confidence interval of 0.82 to 2.24 percentage points. Thus, the validation-set advantage of Random Forest over Decision Tree was statistically supported.

4.2 Cross-validation results

To assess classifier stability without using data reserved for configuration selection or final evaluation, stratified 10-fold cross-validation was conducted exclusively within the 60% training partition. In each iteration, nine folds were used for model fitting and the remaining fold was used for evaluation. Table 2 reports the mean and standard deviation of accuracy, macro-F1, phishing recall, and phishing F1-score across the ten folds.

Random Forest achieved the highest observed mean cross-validation accuracy of 96.64% and macro-F1 of 96.59%, with standard deviations of 0.71% and 0.73%, respectively. XGBoost and Decision Tree produced similar mean macro-F1 values of 95.61% and 95.60%. Naive Bayes achieved the highest mean phishing recall of 99.90%; however, its substantially lower mean macro-F1 of 56.99% indicates poor balance between the phishing and legitimate classes. The training-only cross-validation results were consistent with the validation-set ranking, in which Random Forest also achieved the highest observed macro-F1. These descriptive results supported carrying Random Forest forward to the subsequent validation-based configuration analyses, while the statistical uncertainty of the observed differences was assessed separately.

Table 2. Stratified 10-fold cross-validation results on the training partition

4.3 Compact feature subset analysis

To determine whether feature reduction could preserve detection performance without consulting the final test data, feature importance was first estimated using a Random Forest fitted exclusively to the training partition. The resulting ranking was used to construct subsets containing the top 5, 10, 15, and 20 features, together with the complete 30-feature configuration. A separate Random Forest was trained for each configuration using the training partition and evaluated on the validation partition. Table 3 presents the resulting validation performance.

The validation results show a consistent improvement as additional features were included. The complete 30-feature configuration achieved the highest observed accuracy of 97.24%, macro-F1 of 97.20%, and phishing recall of 96.02%, together with the lowest phishing FNR of 3.98%. The top-20 subset produced similar but slightly lower values, with an accuracy of 97.06%, macro-F1 of 97.02%, phishing recall of 95.71%, and phishing FNR of 4.29%. Therefore, the validation analysis did not provide evidence that reducing the feature set improved predictive performance. The complete 30-feature configuration was consequently selected and frozen for threshold optimization and final test evaluation. The paired comparison did not establish a statistically significant difference between the complete 30-feature model and the top-20-feature model. The complete model correctly classified eight validation instances that the top-20 model misclassified, whereas the top-20 model correctly classified four instances that the complete model misclassified. The two-sided exact McNemar test was not significant (p = 0.3877). The accuracy difference was 0.18 percentage points, with a paired-bootstrap 95% confidence interval from -0.14 to 0.50 percentage points, while the macro-F1 difference was 0.18 percentage points, with a 95% confidence interval from -0.13 to 0.50 percentage points. The complete 30-feature configuration was retained according to the prespecified validation-selection rule, but its small observed advantage over the top-20 subset should not be interpreted as statistically established.

As shown in Figure 1, validation accuracy, macro-F1, and phishing recall generally increased as additional features were included. Although the gains between the top-20 and complete 30-feature configurations were small, the complete feature set achieved the highest observed value for all three metrics and the lowest phishing FNR reported in Table 3. Therefore, all 30 features were retained for the subsequent threshold analysis and independent final test evaluation.

Table 3. Validation-set feature-subset analysis using Random Forest

tu_pian_1.png

Figure 1. Validation performance across Random Forest feature-subset sizes

4.4 Risk-aware threshold optimization

Following validation-based model and feature selection, threshold analysis was conducted using the complete 30-feature Random Forest configuration. The probability of phishing was calculated for the validation partition and thresholds of 0.10 to 0.90 in increments of 0.01 were tested. Two operating points were chosen for comparison with the 0.50 default threshold: a threshold that maximized the macro-F1 and a risk-aware threshold that maximized the phishing recall while keeping the legitimate FPR at or below 5%. These three operating points have been validated and are shown in Table 4. No thresholds were determined or adjusted using final test data.

The validation accuracy and macro-F1 were 97.24% and 97.20%, respectively, while the phishing recall, phishing FNR, and legitimate FPR were 96.02%, 3.98%, and 1.79%, respectively, at the default threshold of 0.50. The highest validation macro-F1 (97.33%) and validation accuracy (97.38%) were obtained with a threshold of 0.52 while phishing recall slightly dropped to 95.81%. The risk-aware threshold 0.29, on the other hand, boosted phishing recall to 98.26% and cut the phishing FNR to 1.74%, while raising the legitimate FPR to 4.95%. These results demonstrate the anticipated operating characteristic: a lower phishing threshold decreases the number of phishing sites that are not detected, and raises the number of false alarms involving legitimate sites.

As shown in Figure 2, increasing the phishing probability threshold reduced the number of websites classified as phishing. Consequently, phishing recall generally decreased, and the phishing FNR increased, while the legitimate FPR declined. The threshold of 0.52 was located near the maximum validation macro-F1, whereas the lower risk-aware threshold of 0.29 increased phishing sensitivity while remaining within the predefined 5% legitimate false-positive-rate constraint. Both operating points were selected exclusively from validation data. After model selection, feature-set selection, and threshold optimization were completed on the training and validation partitions, the resulting Random Forest configuration was evaluated on the untouched 20% final test partition. The model used all 30 structural features. The default threshold of 0.50 and the two validation-selected thresholds of 0.52 and 0.29 were applied without further adjustment. Table 5 reports the independent final test performance of these frozen operating points.

On the untouched test partition, the default threshold of 0.50 achieved the highest accuracy of 97.01% and macro-F1 of 96.97%. The validation-selected macro-F1 threshold of 0.52 produced similar but slightly lower final-test values; it was not readjusted after observing the test results. The risk-aware threshold of 0.29 increased phishing recall from 95.20% to 97.65% and reduced the phishing FNR from 4.80% to 2.35%. This improvement in phishing sensitivity was accompanied by an increase in the legitimate FPR from 1.54% to 4.79% and a reduction in accuracy to 96.29%. These independent results confirm the expected trade-off between reducing missed phishing websites and controlling false alarms without using the test partition for configuration selection.

Table 4. Validation-set threshold selection results for Random Forest

tu_pian_2.png

Figure 2. Validation-set trade-off across thresholds applied to the phishing probability P(y = 0 | x), where class 0 represents phishing websites

Table 5. Independent final-test performance of the frozen Random Forest operating points

4.5 Explainability analysis

Explainability analysis was conducted to identify the most influential structural website features used by the Random Forest model. First, Random Forest impurity-based feature importance was calculated. Then, permutation importance was used to validate whether the same features remained important when their effect on macro-F1 was measured. Table 6 shows the training impurity and post-hoc permutation importance.

As shown in Table 6, the post-hoc final-test permutation analysis ranked URL_of_Anchor first and SSLfinal_State second, whereas the training-derived impurity analysis ranked SSLfinal_State first and URL_of_Anchor second. Across all 30 features, the two rankings showed strong agreement (Spearman ρ = 0.951, p < 0.001), and eight features were shared between their respective top-10 lists. This indicates that the two methods produced broadly consistent, although not identical, explanations. The feature web_traffic ranked third under both methods, while Prefix_Suffix, Links_in_tags, having_Sub_Domain, and Links_pointing_to_page were also prominent. Figure 3 presents the top 10 features according to the post-hoc permutation analysis.

As shown in Figure 3, URL_of_Anchor and SSLfinal_State produced the largest mean decreases in macro-F1 and were clearly more influential than the remaining features under the post-hoc permutation analysis. Their first- and second-place ordering should not be interpreted as a statistically established difference. Their prominence under both impurity and permutation analyses indicates stable model reliance within this dataset; however, these importance values are model and dataset-specific associations and should not be interpreted as causal effects.

Table 6. Training impurity importance and post-hoc final-test permutation importance

tu_pian_3.png

4.6 Discussion

Random Forest has the highest validation accuracy and macro-F1 score, together with high phishing recall among the classifiers observed. Its advantages over XGBoost and Decision Tree were supported by paired exact McNemar tests (p = 0.000303 and p = 0.000038 for Random Forest versus XGBoost and Random Forest versus Decision Tree, respectively) and paired-bootstrap confidence intervals which did not include zero for the accuracy and macro-F1 differences. The remaining classifiers were not tested by paired significance tests, which is why the comparisons are descriptive. The results also demonstrate that the evaluation of phishing detection based on accuracy alone is not sufficient. Although Naive Bayes had a very high phishing recall, it had a low accuracy and a low macro-F1 score, meaning that it misclassified many of the legitimate websites as phishing. This underlines the need to consider balanced and risk-aware metrics, such as macro-F1, phishing recall, phishing FNR and legitimate FPR. The validation-based feature-subset analysis revealed that adding more features led to an increasingly close match of the reduced subsets to the performance of the full feature set. The full 30-feature configuration, however, performed the best in terms of the observed validation accuracy, macro-F1, and phishing recall, and the lowest phishing FNR. The top-20 subset yielded similar results but failed to enhance any of the reported validation metrics. Furthermore, the paired comparison was not significant (McNemar p = 0.3877), and the intervals of the accuracy and macro-F1 differences contained zero, meaning that there was no statistically significant performance difference between the two feature configurations. Hence, all 30 features are kept for the 'Frozen Final' configuration. The reduced-subset results show a trade-off between dimensionality and prediction but do not prove that there is a performance benefit to feature reduction. A security-oriented operating point selection method, which avoids looking at the final test data, was obtained by threshold analysis based on the validation. The threshold of 0.52 was the one that yielded the highest validation macro-F1, while the default threshold of 0.50 yielded a slightly higher macro-F1 on the independent test partition. No threshold was readjusted after the test results were observed. The risk-aware threshold of 0.29 lowered the final-test phishing false negative rate from 4.80% to 2.35% and raised the phishing recall rate from 95.20% to 97.65% while raising the legitimate FPR from 1.54% to 4.79%. The results show the expected trade-off between phishing sensitivity and false alarms, while the evaluation protocol maintained the independence of the final evaluation. Lastly, the training-derived impurity ranking and the post-hoc final-test permutation ranking were highly correlated for all 30 features (Spearman ρ = 0.951, p < 0.001) and had 8 features in common in the top 10. Under both methods, the top two features were URL_of_Anchor and SSLfinal_State, but the ranking of these two features was different. The results show that the model built using the Random Forest algorithm used the anchor-link and SSL-related metrics heavily in the analyzed benchmark and do not imply any cause and effect, nor that they were ordered the same way on more recent external data. One of the drawbacks is the size of the data set and its age. The UCI Phishing Websites dataset was published in 2015 and includes 11,055 examples, with each example having 30 structural features [32]. There were no collection times indicated on the records released, so temporal validation could not be carried out. In addition, the present study did not test an independent external data set with the same feature schema. Therefore, the findings reported here are internal generalizations and should not be assumed to reflect performance in current phishing attacks. A more recent dataset, called PhiUSIIL, comprises 235,795 legitimate and phishing URLs using 54 URL- and webpage-based features. It was not used as an external test of the fitted model as it has a different feature schema from the 30 engineered features used in this study, so direct evaluation of the fitted model would require extracting the same 30 features from its underlying websites. Future work should extend the evaluation protocol to newer independent benchmarks, and to continuously collected (time-stamped) URLs for temporal robustness, as well as to cross-dataset generalizability, concept drift and threshold recalibration.