Reproducible Multiclass Intrusion Detection on UNSW-NB15 Using XGBoost: A Leakage-Safe Pipeline with χ² Feature-Selection Ablations

The increasing complexity of modern networked environments, including enterprise systems, cloud services, industrial networks, and connected infrastructures, has expanded the cybersecurity attack surface. These environments may be exposed to diverse malicious activities, including data exfiltration, botnet communication, reconnaissance, and distributed denial-of-service (DDoS) attacks [1]. Consequently, effective Intrusion Detection Systems (IDSs) are needed to support early threat identification and cyber-risk mitigation [2, 3]. Machine-learning-based IDSs have received considerable attention because they can learn discriminative patterns from network traffic and support the automated identification of malicious activity [4, 5]. Ensemble-learning methods, including Random Forest and eXtreme Gradient Boosting (XGBoost), are widely used for structured network-traffic data because they can model nonlinear relationships among heterogeneous flow features [6, 7]. However, the reliability of machine-learning-based IDS evaluation can be affected by class imbalance, limited analysis of minority attack categories, and inconsistent experimental procedures [8]. Furthermore, feature-selection methods are frequently applied to reduce dimensionality, but their effect on multiclass detection performance should be evaluated using transparent and leakage-controlled experimental protocols [9]. To address these evaluation challenges, this paper develops a reproducible and leakage-safe multiclass intrusion-detection pipeline based on XGBoost and evaluates it on the UNSW-NB15 benchmark. The evaluation supports cyber-risk mitigation by identifying attack families that remain difficult to detect and by showing where classification errors are concentrated. Specifically, we compare a full-feature XGBoost model with $\chi^2$-based top-k feature-selection variants under the same fixed stratified split, train-only preprocessing protocol, and evaluation settings. In addition to overall accuracy, we report macro-averaged and per-class metrics together with confusion-matrix-based error analysis to characterize performance differences between frequent and rare attack families. This framing positions the work as a controlled benchmark and ablation study for multiclass network intrusion detection on UNSW-NB15. Accordingly, the findings of this study are limited to the UNSW-NB15 network-traffic setting and do not constitute direct validation on IoT-specific traffic or resource-constrained IoT platforms. Existing XGBoost- and feature-selection-based intrusion-detection studies often emphasize overall predictive performance or optimize a specific model configuration. In contrast, the present study keeps the data split, preprocessing procedure, model configuration, and evaluation protocol fixed while examining whether $\chi^2$-based feature reduction changes multiclass detection behavior. The contribution therefore lies in providing a controlled and reproducible comparison between full-feature and reduced-feature XGBoost models, with particular attention to macro-level and minority-class performance under class imbalance. Our contributions can be summarized as follows:

• We develop a reproducible and leakage-safe multiclass intrusion detection pipeline for UNSW-NB15 using XGBoost, with all preprocessing steps fitted on the training split only to ensure fair evaluation.
• We conduct a controlled $\chi^2$ feature-selection ablation by comparing multiple top-k settings against the full-feature model under identical data split, preprocessing, and training conditions.
• We provide a class-imbalance-aware evaluation through macro-averaged metrics, per-class Precision/Recall/F1, and row-normalized confusion-matrix analysis, highlighting the differing behavior of head and tail attack classes.
• We show that the full-feature XGBoost model provides the strongest overall performance on the evaluated split, while $\chi^2$ reduction offers a transparent dimensionality-reduction baseline without surpassing the full model.

The remainder of the paper is structured as follows. Section 2 reviews related work on machine-learning-based IDSs. Section 3 outlines the proposed framework and feature selection methodology. Section 4 describes the experimental design, including datasets, model configurations, and evaluation metrics. Section 5 presents and discusses the results. Section 6 concludes the paper and outlines future research directions.

This section details the dataset, data partitioning, preprocessing pipeline, feature‑selection ablation, model configurations, and evaluation protocol. All steps are designed for reproducibility and no leakage, implemented as scikit learn/XGBoost pipelines.

3.1 Data and split

We use the UNSW‑NB15 dataset, which comprises realistic background traffic and nine contemporary attack families (Analysis, Backdoor, DoS, Exploits, Generic, Reconnaissance, Shellcode, Worms) plus Normal, yielding a 10‑class problem. To perform a controlled within-study comparison between the full-feature XGBoost model, the Random Forest baseline, and the $\chi^2$ feature-selection variants, the official UNSW-NB15 training and testing files were pooled and then divided using a single stratified hold-out split (80% training and 20% testing; random seed = 42). Stratification preserves the class proportions of the pooled dataset in both partitions, including the minority attack families.

The pooled corpus contains 257,673 network flows after duplicate detection and basic consistency checks. Class distributions are imbalanced, with Normal and Generic representing the largest classes and Backdoor and Worms representing rare classes; therefore, macro-averaged and per-class metrics are reported in addition to overall accuracy.

This pooled-split protocol was selected to ensure that all evaluated configurations use identical training and testing conditions. Because it differs from the official UNSW-NB15 train/test partition, the reported results should be interpreted as a controlled internal evaluation and should not be compared directly, in numerical terms, with studies that use the official split.

Table 1 reports the class counts and percentages before and after cleaning. These statistics describe the combined official partition that underlies all subsequent experiments in this paper.

Table 1. Class distribution of the merged UNSW-NB15 dataset before and after cleaning (TRAIN+TEST = 257,673 flows)

Data-quality checks, including duplicate detection and label-consistency checks, did not remove any records. Therefore, the cleaned distribution equals the raw merged distribution. Table 1 shows a strongly imbalanced distribution (Normal and Generic dominate, Shellcode and Worms are rare). Accordingly, we will report per-class Precision/Recall/F1 and Macro-F1 (class-averaged) in addition to Accuracy. The subsequent train/test evaluation preserves these proportions via stratification (seed = 42).  The merged dataset used in this study contains 45 columns: 42 predictor variables, the record identifier (id), and two target-related variables, namely attack_cat for multiclass classification and label for binary attack-versus-normal classification. The predictor variables describe packet statistics, protocol-level behavior, and flow-related characteristics. The UNSW-NB15 file of official feature list was used to assign the column headers manually to make sure that the features and labels matched well. No rows were removed for missing values because the final merged table contained no missing entries; categorical variables were encoded within the train-only preprocessing pipeline.

Figure 1 summarizes the pooled stratified evaluation workflow. Leakage-safe preprocessing and model fitting are performed on the training split only. The workflow evaluates XGBoost variants, Logistic Regression, Random Forest, Extra Trees, and an optional $\chi^2$ feature-selection branch, using classification metrics, Macro-ROC-AUC, Macro-PR-AUC, and confusion matrices, with bootstrap confidence intervals for the primary XGBoost model.

image003.png

Figure 1. Proposed leakage-safe workflow for multiclass intrusion detection on UNSW-NB15

3.2 Pre‑processing

We follow a reproducible, train-only pre-processing pipeline built with scikit-learn. After concatenating the official UNSW-NB15 TRAIN and TEST partitions (total 257,673 flows; see Section 3.1), we checked for exact duplicate rows and verified that the final table contained no missing values. No records were removed during these checks. After excluding the record identifier (id), the binary target-related variable (label), and the multiclass target (attack_cat), the predictor matrix contains 42 raw features: 39 numerical features and 3 categorical features. One-hot encoding expands the training-derived predictor space to 195 transformed features. All transformers are fit on the training split only and then applied to the held-out test split to avoid leakage.

For the main model (XGBoost, no feature selection), numeric features are used as is (no standardization is required for tree ensembles), and the OHE outputs are passed directly to the classifier. For the feature-selection ablation ($\chi^2$/SelectKBest), we first apply MinMax scaling to the numeric features to satisfy the non-negativity assumption of $\chi^2$ (OHE outputs are already non-negative); we then compute $\chi^2$ scores on the training split only and evaluate predefined top-k configurations(k∈{80,100,120,160}). The selected columns are concatenated (numeric-scaled + OHE) and used to train XGBoost with the same hyperparameters as the full-feature model. At inference time, the fitted pre-processor and the $\chi^2$ selector are applied to the test split in the same order. We keep the attack_cat label in a 10-class setting: Normal plus {Fuzzers, Analysis, Backdoor, DoS, Exploits, Generic, Reconnaissance, Shellcode, Worms. Class imbalance is addressed through macro-averaged and per-class evaluation. In addition, an imbalance-aware sensitivity experiment trains XGBoost using inverse-frequency sample weights calculated exclusively from the training labels; no resampling is applied. Random seeds are fixed (seed = 42) for the stratified split and model initialization. The entire pre-processing is scripted so that the fitted encoders/scalers/selectors can be serialized and reused for exact reproducibility.

3.3 Feature‑selection ablation ($\chi^2$)

We treat feature selection as a controlled ablation, evaluating whether reducing dimensionality improves or preserves performance relative to the full-feature model. Because the $\chi^2$ statistic measures the univariate dependence between a non-negative feature and the class label, we first apply MinMax scaling to numeric features and one-hot encoding (OHE) to categorical features; both steps are performed inside a scikit-learn pipeline that is fit on the training split only and then applied to the test split, preventing any form of test leakage. The $\chi^2$ scores are then computed on the training data only over the transformed feature matrix, and the top-k columns are selected with SelectKBest($\chi^2$).

As an exploratory view of the ranking produced by $\chi^2$, Figure 2 shows the top 20 features by $\chi^2$ score computed on the training split only (after MinMax+OHE). This bar chart is for illustration and analysis of discriminative signals; it is not used to pick the final K. For the ablation, we evaluate k∈{80,100,120,160}, representing moderate-to-high feature-retention levels within the 195-dimensional transformed feature space. These values were selected to examine whether retaining progressively larger subsets preserves multiclass performance relative to the full-feature model. For each k, we (i) fit the MinMax+OHE transformers on the training split, (ii) compute $\chi^2$ on the training split and keep the top- k columns, and (iii) train XGBoost with the same hyperparameters as the full-feature model. At inference time, the fitted pre-processor and the $\chi^2$ selector are applied to the held-out test set in the same order. Aggregate and per-class performance for this ablation are reported in Section 4.4, as well as the efficiency comparison. The $\chi^2$ ranking computed on the training split after Min–Max scaling and one-hot encoding is shown in Figure 2 for descriptive analysis only. The candidate k values were predefined, and the full-feature model remains the reference configuration for comparison.

As shown in Figure 2, the highest $\chi^2$ scores indicate strong univariate association for several flow-derived and protocol/service features on the training split. Because $\chi^2$ is univariate, we treat this plot as descriptive only; model selection uses the predetermined K sweep and the held-out test results reported later.

image004.png

Figure 2. Top-20 transformed features ranked by $\chi^2$ score on the training split after Min–Max scaling and one-hot encoding

3.4 Models and hyperparameters

We evaluate four classifiers under the same leakage-safe preprocessing and stratified evaluation protocol:

• XGBoost (primary). Gradient-boosted decision trees are trained using the multiclass probability objective (multi:softprob). To maintain comparability, the following hyperparameters are fixed across the XGBoost experiments: n_estimators = 300, max_depth = 6, learning_rate = 0.10, subsample = 0.80, colsample_bytree = 0.80, min_child_weight = 1, reg_lambda = 1.0, tree_method = 'hist', eval_metric = 'mlogloss', and random_state = 42. For the unweighted XGBoost reference model, class imbalance is not reweighted during training and is assessed through macro-averaged and per-class evaluation metrics. To examine the effect of class imbalance during training, we additionally evaluate a class-balanced XGBoost variant. This model uses the same preprocessing pipeline and XGBoost hyperparameters as the unweighted reference model, but applies inverse-frequency sample weights derived from the training-set class distribution. The purpose of this experiment is to assess whether improved minority-class recall is obtained at the cost of overall detection performance.
• Random Forest (baseline). A bagging-based tree-ensemble baseline implemented with scikit-learn and configured with n_estimators = 200, max_features = 'sqrt', bootstrap = True, and n_jobs = -1.
• Extra Trees. An Extremely Randomized Trees classifier is included as an additional randomized tree-ensemble baseline. It is configured with n_estimators = 200, max_features = 'sqrt', bootstrap = False, and n_jobs = -1.
• Logistic Regression. A multinomial logistic-regression classifier is included as a linear baseline. Numerical features are standardized within the training pipeline, whereas categorical features are one-hot encoded. The model uses the L-BFGS solver with a maximum of 1,000 iterations.
• All transformers (OHE/MinMax) and, when applicable, the $\chi^2$ selector are fit exclusively on the training set. Pipelines are implemented using scikit‑learn’s Pipeline/ColumnTransformer to guarantee correct ordering and prevent leakage. We fix random seeds (42) for data partitioning and model initialization. For interpretability and error analysis, we produce row‑normalized confusion matrices (diagonal equals per‑class recall) alongside standard metrics.

3.5 Evaluation metrics and reporting protocol

In section 3.1, we defined the held-out test set that is used to evaluate models, and we adopted a consistent train-only pipeline. As UNSW-NB15 is a class-imbalanced dataset, we include macro-averaged numbers as well as per-class numbers.

(1) Primary metrics

We report on Accuracy, macro-Precision, macro-Recall and macro-F1. Macro-averaged measures are the simple average of scores on the individual classification tasks, and so are the same for majority and minority classes, for a K-class classification task. In addition, we report Weighted-F1 which weighs the F1 score of each class by its class support, giving a performance measure that reflects the distribution of classes. Precision is the percentage of the instances correctly predicted as class c among all instances predicted as class c; Recall is the percentage of the instances correctly predicted as class c among all instances of class c. Averaging across all classes of the harmonized averages of Precision and Recall for class c yields the F1-score. Macro-averaging is then used to compute the averages of the per-class F1 scores.

(2) Confusion matrices

We include row-normalized confusion matrices (each row sums to 1.0), so the diagonal equals per-class recall. This view highlights where errors flow from each true class into predicted classes, which is essential for interpreting minority-class behavior under imbalance.

(3) Ranking metrics

In addition to threshold-based metrics, we compute Macro-ROC-AUC using the one-vs-rest formulation and Macro-PR-AUC using one-vs-rest average precision. Macro-PR-AUC is particularly informative under class imbalance because it reflects the precision–recall trade-off across attack categories.

(4) Decision rule / thresholds

Multiclass predictions are obtained by argmax over class probabilities (multi: softprob for XGBoost). We do not apply post-hoc threshold tuning or probability calibration in the main results; this keeps comparisons fair across models/ablations.

(5) Statistical confidence

To quantify uncertainty for the primary full-feature XGBoost model, we compute 95% confidence intervals for Accuracy, Macro-F1, and Macro-PR-AUC using a stratified percentile bootstrap on the held-out test set with 1,000 resamples.

(4) Reporting conventions

We report percentages with two decimal places and express ablation differences in percentage points relative to the full-feature XGBoost reference model.

The comparative evaluation shows that XGBoost provides the strongest balance of overall and macro-level performance among the evaluated linear, bagging-based, and boosted-tree models. This finding supports the use of gradient-boosted trees for structured intrusion-detection data, where interactions among network-flow variables can be important. Unlike evaluations that focus mainly on aggregate accuracy, the present analysis combines macro-level metrics, per-class results, Macro-PR-AUC, and bootstrap confidence intervals to provide a clearer view of performance under class imbalance.

The per-class analysis indicates a clear head–tail pattern. Frequent categories are detected more reliably than rare attack families, whereas Analysis, Backdoor, DoS, Shellcode, and Worms remain more difficult to classify consistently. This confirms that high weighted performance should not be interpreted as uniformly strong detection across all attack categories. The class-balanced XGBoost sensitivity experiment further illustrates this trade-off: increasing the influence of minority classes improves recall for several rare categories but reduces aggregate performance and Macro-F1.

The $\chi^2$ ablation provides a complementary result. Reducing the transformed feature space from 195 to 80 features lowers training time, but does not improve Macro-F1, end-to-end inference time, or serialized pipeline size. Thus, $\chi^2$ selection is useful mainly when reduced feature dimensionality or lower training cost is a practical priority, rather than as a replacement for the full-feature configuration. The findings should be interpreted within the study design. The evaluation uses one pooled stratified split of UNSW-NB15 and therefore represents a controlled within-study comparison rather than direct replication of the official dataset partition. In addition, cross-dataset generalization, adversarial robustness, and deployment on constrained hardware were not evaluated. Future work should examine class-sensitive learning, calibrated class-specific decision rules, and cross-dataset evaluation to determine whether minority-class detection can be improved without a substantial loss of overall performance.