Generalization-Aware Evaluation of a Regularized Multi-Layer Perceptron for Binary Network Intrusion Detection on NSL-KDD

The proliferation of interconnected information systems, cloud services, edge platforms, and internet-connected devices has led to substantial growth in network traffic and a wider variety of cyber threats. Network intrusion detection systems (NIDSs) are therefore important analytical tools for identifying traffic patterns associated with malicious activity. Signature-based systems compare observed traffic with known attack patterns and generate alerts when those patterns are detected, whereas anomaly-based systems learn or estimate normal behavior and report deviations from that baseline. The latter can assist in detecting previously unseen behavior, but their effectiveness depends on the representativeness of the training data, the false-alarm rate, and performance stability under changes in traffic distribution [1, 2].

Machine learning and deep learning have broadened the design space for anomaly-based intrusion detection by enabling nonlinear feature interactions to be learned from high-dimensional network data. Recent studies have investigated autoencoder-assisted deep networks, recurrent and hybrid architectures, cost-sensitive learning, self-supervised graph representations, and sequential deep models [3-9]. These advances demonstrate the flexibility of data-driven NIDSs, but they also show that reported accuracy values are highly sensitive to dataset age, class balance, preprocessing choices, and evaluation protocol.

A connection-level classifier should be considered one component of a layered control architecture rather than a replacement for signature-based, session-level, or cryptographic controls. Its usefulness depends not only on classification accuracy, but also on attack recall, false-positive behavior, computational cost, calibration, and robustness to out-of-distribution traffic. Accordingly, this study emphasizes evaluation discipline and generalization analysis rather than claims of production-deployment readiness.

NSL-KDD continues to be used as a historical benchmark because it reduces many of the duplicate-record problems present in KDD Cup 1999 and facilitates reproducible comparisons [10]. However, its traffic behavior and attack patterns reflect an earlier generation of network environments. More recent traffic characteristics and attack scenarios are represented in contemporary benchmarks such as CICIDS-2017 [11] and UNSW-NB15 [12], while recent research has increasingly emphasized cross-environment generalizability and robustness in addition to within-dataset accuracy [9, 13-15]. This distinction is central to the present evaluation.

The study does not claim a novel neural-network architecture. Rather, it reports four evaluation-oriented contributions: (1) a reproducible regularized multi-layer perceptron (MLP) baseline that combines established batch normalization (BN), LeakyReLU, L2, and dropout components; (2) a leakage-controlled training and evaluation protocol in which learned transformations are fitted to training data only; (3) threshold-sensitivity and computational-cost analyses; and (4) a direct contrast between confirmatory multi-seed KDDTest+ results and exploratory single-seed ablation and imbalance-handling comparisons.

NIDSs are commonly described along two dimensions: signature-based versus anomaly-based detection, and host-based versus network-based monitoring. Signature-based NIDSs perform well when reliable signatures are available for known attacks, but their effectiveness is limited when attack signatures change or when a new attack has no established signature. In contrast, anomaly-based methods model typical or previously observed behavior and identify deviations from that baseline, which can improve sensitivity to novel behavior but may also increase false-positive alerts. This trade-off supports the use of precision, recall, F1-score, receiver operating characteristic (ROC) measures, and threshold-dependent error analysis rather than accuracy alone [1].

Deep models are helpful for learning nonlinear representations from mixed numerical and categorical network features, though there is no guarantee that more complex models will generalize better. NSL-KDD is a tabular dataset at the connection level, instead of a raw sequence of packets, which means that it has a relatively straightforward baseline with a limited number of architectural assumptions, provided by a feed-forward MLP. Since regularization is applied to control the fitting behavior in KDDTrain+ in this study, it cannot be assumed that these choices lead to better performance on a shifted test distribution.

3.1 System overview and design rationale

The system consists of four steps: (i) data preparation and binary labeling, (ii) categorical encoding and numerical standardization, (iii) training of the regularized MLP, and (iv) testing on both KDDTrain+-derived partitions and the official KDDTest+ set. All preprocessing parameters are learned from the training data and then applied unchanged to the validation and test data. The implementation uses Python 3.9, TensorFlow 2.10, scikit-learn 1.0.2, and pandas 1.3.5. The reference run uses seed 42, while five independent random-seed runs are used to obtain the main KDDTest+ generalization statistics.

A feed-forward MLP was chosen because each NSL-KDD connection is represented as a fixed tabular feature vector rather than as an ordered sequence of raw packets. Recurrent architectures introduce a temporal inductive bias that is not directly aligned with this representation, while convolutional mappings impose spatial structure on heterogeneous features. The MLP is therefore used as a deliberately conventional baseline against which the effects of regularization and the evaluation protocol can be examined without introducing claims of a new architectural mechanism.

3.2 Dataset description

NSL-KDD is an improved benchmark based on the KDD Cup 1999/DARPA traffic lineage [10]. There are 41 input attributes for each record, an attack label, and a difficulty indicator. KDDTrain+ includes 125,973 records from the following attack families: denial-of-service (DoS), Probe, Remote-to-Local (R2L), and User-to-Root (U2R). The official KDDTest+ set is kept unchanged and contains 22,544 records for generalization analysis. The dataset statistics and partitioning used in this study are summarized in Table 1.

The experiments use a binary target in which all attack subcategories are grouped into a single attack class and normal traffic forms the second class. This formulation focuses the analysis on attack-versus-normal discrimination and allows the effect of distribution shift to be interpreted directly through attack precision and recall. It does not support conclusions about the model's ability to identify individual attack families.

Table 1. NSL-KDD dataset statistics and partitioning

3.3 Feature characterization

The 41 NSL-KDD input attributes can be grouped into basic traffic, content, time-window, and host-based categories, as summarized in Table 2. Basic features describe individual connections; content features capture indicators such as failed logins or compromised states; time-window features summarize recent connection activity; and host-based features describe activity associated with destination hosts. These heterogeneous feature types require explicit categorical encoding and standardized numerical scaling before MLP training.

Table 2. NSL-KDD feature categories and representative attributes

3.4 Preprocessing pipeline

Among the 41 input attributes, protocol_type, service, and flag are nominal variables. They are one-hot encoded so that no artificial ordinal relationship is imposed between categories, producing 122 post-encoding input dimensions in the implementation. The difficulty-level field is excluded because it is a benchmark annotation rather than an observable traffic attribute intended for prediction.

Only training-set statistics are used to standardize numerical features: x' = (x − μ_train) / σ_train. The fitted scaler is then applied without modification to the validation set, the internal test set, and the official KDDTest+ set. This separation prevents test information from influencing feature transformation. Binary labels are encoded consistently across the pipeline, with attack traffic defined as the positive detection target for precision, recall, F1-score, and threshold-dependent confusion matrices.

For the training labels, class weights are calculated as w_c = n_samples / (n_classes × n_c), where n_samples denotes the total number of training samples, and n_c denotes the number of samples in class c. The resulting weights are approximately 1.0743 for attack traffic and 0.9353 for normal traffic. Class weighting modifies each class's contribution to the loss without creating additional training instances, which is relevant because class-imbalance treatment can substantially affect minority-attack detection [7, 16].

3.5 Data partitioning

KDDTrain+ is divided in a stratified manner into training, validation, and internal hold-out subsets using nominal proportions of 70/15/15. The internal hold-out is not used for model fitting or hyperparameter tuning, while the validation subset is used by callbacks for model selection. The official KDDTest+ set is never included in this partitioning process and serves as the primary measure of generalization beyond the KDDTrain+ distribution.

3.6 Class imbalance treatment

Class imbalance may cause a classifier to become biased towards the more common class, and may make accuracy itself a limited measure of quality of detection. Recently, this issue has been tackled by resampling, cost-sensitive learning, ensemble techniques, and algorithm-level weighting [7, 16]. The imbalance is moderate in the present binary KDDTrain+ formulation, and class weighting is kept without any synthetic samples to reduce frequency-driven bias.

Therefore, in the principal training configuration, class weighting is used. The weights are directly applied to the loss function, such that errors from the less frequent class have more impact in the optimization process. A single-seed comparison with SMOTE is reported later as an exploratory, single-seed diagnostic and is not repeated across multiple seeds, so it is not used to make a general superiority claim for class weighting.

3.7 Implementation environment

All experiments were performed in Python 3.9 using TensorFlow 2.10 (Keras), scikit-learn 1.0.2 for preprocessing and evaluation, and pandas 1.3.5 for data manipulation. The Adam optimizer was used with the parameters listed in Table 3, a batch size of 256, and early stopping based on validation loss. All computations were performed in a CPU-only environment with an Intel Core i7 processor and 16 GB of RAM. Measured training and inference times are implementation-specific and are not interpreted as evidence of production throughput or deployability.

Reproducibility controls included explicit NumPy and TensorFlow seeds for the reference run, a fixed random state for the reference data split, training-only fitting of the StandardScaler, and checkpointing of the model weights associated with the lowest validation loss. The complete training procedure was repeated using five independent seeds, and the mean and standard deviation of the KDDTest+ metrics were calculated so that the primary official-test results reflect training variability rather than a single initialization.

Table 3. Training hyperparameter configuration

5.1 Training dynamics on KDDTrain+-derived partitions

Training and validation accuracy and binary cross-entropy loss for the reference run are shown in Figure 2. The curves converge rapidly and remain close after the initial epochs, indicating stable fitting on the KDDTrain+-derived partitions. These curves are useful for assessing within-distribution training behavior, but they do not demonstrate that the classifier will generalize to the different KDDTest+ distribution.

tu_pian_2.png

The associated checkpoint with the minimum validation loss was restored at early stopping. The small separation between training and validation curves implies that there is little divergence between the curves for this run; however, the significant difference between training and official-test performance seen in Section 5.2 warns of the dangers of using internal learning curves as a proxy for performance on distribution-shift evaluation.

5.2 Primary generalization results on the official KDDTest+ set

The principal result of this study is performance on the untouched official KDDTest+ set across five independent random seeds. Mean accuracy was 78.14% ± 0.46%, precision was 92.37% ± 0.32%, recall was 67.15% ± 0.81%, F1-score was 77.76% ± 0.57%, and area under the receiver operating characteristic curve (AUC-ROC) was 0.8767 ± 0.0162. Table 5 places these values beside the internal hold-out results to make the generalization gap explicit. The most consequential result is the recall of 67.15%: under the official distribution shift, approximately one third of attack records are not detected at the evaluated operating point, despite much stronger precision.

Table 5. Official KDDTest+ generalization versus the internal KDDTrain+-derived hold-out

Note: AUC-ROC = area under the receiver operating characteristic curve.

5.3 Secondary within-distribution performance

The internal hold-out, which is derived from KDDTrain+, produced 99.43% accuracy, 99.45% precision, 99.50% recall, 99.47% F1-score, and an AUC-ROC of 0.9994. These values indicate strong discrimination within the same general source distribution as the training data, but they are not interpreted as measures of the model's generalization performance. Relative to the five-seed KDDTest+ results, accuracy decreases by 21.29 percentage points and recall decreases by 32.35 percentage points. The magnitude of these differences demonstrates that the internal-split evaluation substantially overestimates performance under the official benchmark distribution shift.

Variation across random seeds does not appear to explain the observed generalization gap because the between-seed variability is small relative to the performance difference between the internal hold-out and KDDTest+. This indicates that the gap is not primarily attributable to random initialization. Under this benchmark shift, recall decreases substantially while precision remains comparatively high, indicating that missed attacks are a more prominent failure mode than false alarms.

Therefore, the official KDDTest+ values, rather than the 99.43% internal accuracy, are used as the primary basis for the interpretation, limitations, and conclusion of this manuscript.

This separation also limits deployment inferences even when a model performs strongly on KDDTrain+. If a substantial proportion of attacks is missed under a different traffic distribution, strong internal performance cannot establish operational effectiveness. The present results therefore support a methodological conclusion about benchmark generalization, not a conclusion about effective real-world operation.

5.4 Secondary per-class analysis on the internal hold-out

The precision, recall, and F1-score values for the KDDTrain+-derived internal hold-out are presented in Table 6. The attack and normal classes show similar performance within this distribution. These values describe internal behavior only and should be interpreted alongside, rather than in place of, the lower official KDDTest+ recall reported in Section 5.2.

Table 6. Secondary per-class report on the internal hold-out (n = 18,896)

5.5 Internal hold-out confusion matrix

The confusion matrix in Table 7 and Figure 3 summarizes the reference internal hold-out at the default threshold. Among the reported samples, 8,739 normal records and 10,050 attack records were classified correctly, while 56 normal records were flagged as attacks and 51 attack records were missed. The resulting 107 errors (0.566% of the internal hold-out) characterize only the KDDTrain+-derived distribution and must not be extrapolated to KDDTest+.

tu_pian_3.png

Table 7. Confusion matrix on the internal hold-out

Note: TP = True Positive, TN = True Negative, FP = False Positive, FN = False Negative.

The high internal accuracy reflects the small numbers of False Positives and False Negatives within the KDDTrain+-derived hold-out, but it does not resolve the generalization issue. Consequently, the subsequent discussion emphasizes official-test recall rather than the internal confusion matrix because KDDTest+ reveals a substantially greater missed-attack burden.

5.6 Internal receiver operating characteristic diagnostic

Figure 4 shows the ROC curve for the internal hold-out. It achieves an AUC-ROC of 0.9994 for records sampled from the KDDTrain+-derived distribution, indicating strong ranking discrimination within this distribution. This value should not be interpreted as evidence of equivalent discrimination under distribution shift; across five KDDTest+ seeds, the mean AUC-ROC is 0.8767 ± 0.0162.

The difference between the two AUC values further supports the primary evaluation result: discrimination on the official test set is substantially lower than within the KDDTrain+-derived hold-out. The internal ROC curve is therefore used only as a diagnostic of within-distribution fit and not as evidence of general detection capability.

tu_pian_4.png

5.7 Internal precision-recall diagnostic

Figure 5 shows the precision-recall curve for the internal hold-out. The high area under the precision-recall curve (AUPRC) reflects separation within the KDDTrain+-derived distribution, but it should not be interpreted as evidence of equivalent precision-recall behavior on KDDTest+. On the official test set, mean precision remains high at 92.37%, whereas mean recall falls to 67.15%, revealing a materially different operating trade-off.

tu_pian_5.png

This difference is especially important for intrusion detection because a high precision value can coexist with an operationally important missed-attack rate. For that reason, the generalization analysis emphasizes recall and F1-score in addition to accuracy and AUC.

5.8 Internal prediction-score distribution

Figure 6 shows the distribution of reference-run prediction scores on the internal hold-out. Most internal records are concentrated near the extremes of the score range, producing limited overlap around the 0.5 decision threshold and explaining the small number of internal errors. This concentration is a property of the KDDTrain+-derived hold-out and should not be interpreted as evidence that scores are similarly separated for attack patterns represented differently in KDDTest+.

tu_pian_6.png

5.9 Decision-threshold sensitivity on official KDDTest+

Threshold behavior is interpreted primarily using KDDTest+ because it is more directly related to the generalization question examined in this manuscript. Figure 7 is retained as a secondary illustration of the internal hold-out, whereas the substantive evidence regarding operating-point sensitivity is provided by the official-test threshold sweep.

For the reference KDDTest+ run (seed 42), lowering the threshold to t = 0.1 produced 91.9% precision, 73.3% recall, and 81.6% F1-score; the default t = 0.5 produced 92.5% precision, 68.2% recall, and 78.5% F1-score; and t = 0.9 produced 93.0% precision, 62.4% recall, and 74.7% F1-score. Overall, precision changed only slightly across the evaluated thresholds, whereas recall decreased steadily. Thus, increasing the threshold can reduce false-positive rates but also increases the number of missed attacks. Across the evaluated KDDTest+ thresholds, no operating point eliminated the recall deficit associated with the distribution shift.

These results indicate that the distribution-shift problem cannot be resolved through threshold selection alone. Lower thresholds provide higher attack sensitivity at the cost of more False Positives, whereas higher thresholds slightly increase precision and reduce False Positives but further decrease attack sensitivity.

tu_pian_7.png

Accordingly, no production operating threshold is recommended on the basis of these results. Any operational threshold would need to be calibrated using current, representative traffic and explicit cost functions for missed attacks and analyst alert burden.

The threshold analysis therefore demonstrates a genuine precision-recall trade-off under KDDTest+, reinforcing the need to interpret performance in relation to the intended operating objective.

5.10 Recent literature and protocol-aware comparison

Table 8 places the present work within recent IDS research without ranking heterogeneous studies solely by accuracy. The selected studies illustrate current directions in hyperparameter-optimized deep learning, multi-dataset validation, cost-sensitive learning, self-supervised anomaly detection, generalizability analysis, and robustness to out-of-distribution or adversarial traffic [3, 5, 7-9, 13-15]. Because the datasets, class definitions, and evaluation partitions differ, their reported scores are not directly comparable with the present KDDTest+ result. The appropriate comparison is methodological: recent work increasingly emphasizes robustness and modern traffic, while this study quantifies how a conventional regularized MLP degrades when NSL-KDD evaluation moves from an internal split to the official test distribution.

Table 8. Protocol-aware context from recent intrusion-detection studies

Note: DAE = Denoising Autoencoder, DNN = Deep Neural Network, MLP = multi-layer perceptron, HPO = Hyperparameter Optimization, NIDS = Network intrusion detection system.

5.11 Computational characterization

The evaluated MLP contains approximately 76,545 parameters. In the reported CPU-only environment, batched inference over KDDTest+ required a mean of 23.6 ms per 1,000 preprocessed records. This measurement indicates that the neural-network forward pass is computationally modest in the experimental setup. It does not establish end-to-end production throughput because packet capture, feature extraction, data transfer, concurrent workloads, security orchestration, and hardware variability were not included in the timing experiment. The latency result is therefore reported as a computational characterization, not as a deployment claim.

5.12 Internal error inspection

There were 107 errors in the reference internal hold-out. False-positive and false-negative cases were concentrated around the decision boundary, while inspection of the prediction scores showed that correctly classified cases were distributed more broadly. This observation helps characterize the internal confusion matrix but is of secondary importance because the official KDDTest+ evaluation exhibits a substantially higher missed-attack rate.

This internal error inspection does not imply that any specific attack family can be detected reliably or that the model's errors are limited to a small set of ambiguous examples. Attack-family-level evaluation should be conducted on the official test set and validated using contemporary datasets to provide a more informative failure analysis.

5.13 Generalization and distribution-shift analysis

Two questions must be distinguished: does the model fit the KDDTrain+-derived partitions consistently, and does the resulting decision function generalize to the official KDDTest+ distribution? The internal curves and metrics indicate consistent within-distribution fitting, but they are insufficient to establish external generalization. Across five seeds, official KDDTest+ accuracy is 78.14% ± 0.46%, and AUC-ROC is 0.8767 ± 0.0162, compared with 99.43% accuracy and 0.9994 AUC-ROC on the internal hold-out. Most importantly, attack recall decreases from 99.50% internally to 67.15% ± 0.81% on KDDTest+. This reduction of approximately 32 percentage points is the clearest manifestation of distribution shift and represents the principal empirical finding of the manuscript, consistent with broader concerns regarding the generalizability of ML-based NIDSs [9, 13].

5.14 Exploratory single-seed ablation of regularization components

The ablation analysis is explicitly exploratory because each configuration was evaluated in a single run. Under this limitation, the full L2 + dropout + BN configuration achieved 78.7% accuracy and 0.893 AUC on KDDTest+; removing L2 yielded 78.6% and 0.896; removing dropout yielded 81.2% and 0.877; removing BN yielded 78.9% and 0.879; and disabling all three regularizers yielded 80.4% and 0.910. These single-seed differences do not establish causal contributions, stable component rankings, or statistical superiority. They indicate only that the relative ordering can change under KDDTest+ distribution shift and therefore motivate a future multi-seed factorial ablation. No conclusion in this manuscript attributes the five-seed generalization performance to any individual regularization component.

5.15 Exploratory class-imbalance comparison

The comparison among no imbalance treatment, class weighting, and SMOTE was also conducted as a single-seed diagnostic and is therefore exploratory. In that run, the respective KDDTest+ accuracy/AUC values were 78.3%/0.865, 80.6%/0.889, and 78.4%/0.873. Although class weighting produced the highest accuracy and AUC in this particular run, the experiment does not support a general claim that class weighting is superior to SMOTE. Multiple random seeds and, ideally, contemporary datasets are required before drawing a stable conclusion about the imbalance-handling strategy.

5.16 Practical interpretation and benchmark limitations

Claims of advanced detection capability or production readiness are not warranted on the basis of the official KDDTest+ results. The mean recall is 67.15%, indicating that a substantial proportion of attacks remains undetected under the benchmark distribution shift. Therefore, the internal scores should not be used for production false-negative-rate estimation, security-level certification, or capacity planning. Inference latency was measured only in a controlled environment and reflects the model's forward pass rather than end-to-end operational performance.

NSL-KDD is also a dated benchmark that cannot represent contemporary encrypted traffic flows, application-layer behavior, current threat morphologies, or organization-specific network dynamics. Validation on contemporary datasets, such as CICIDS-2017 [11] and UNSW-NB15 [12], using repeated seeds, cross-dataset tests, attack-family analysis, and adversarial or out-of-distribution evaluation [9, 13, 15] is required to support stronger deployment conclusions. Until such validation is completed, the model should be regarded as a reproducible baseline for studying generalization rather than as a validated operational IDS.

The author would like to thank the research community for making benchmark datasets and reproducible evaluation tools available for comparative analysis in network security.