Exploiting Anomalies with Data Mining Techniques to Enhance Cloud Security

Network security has become crucial due to internet and communication technology improvements in the previous decade [1]. To protect the network and its assets online, firewalls, antivirus software, and Intrusion Detection System (IDS) are used. Network Intrusion Detection Systems (NIDSs) analyze network traffic for malicious or suspicious activity [2]. Hussein [3] introduced IDS, and since then, many network security IDS products have been developed. But recent technical advances have increased network size and the amount of applications employed by network nodes [3, 4]. Thus, these nodes generate and exchange a lot of valuable data.

Due to new attack methods, including versions of old and novel attacks, protecting this data and network nodes is difficult [4]. Nearly every node in a network is susceptible to security threats [5]. For instance, the security of data nodes is critical for organizations. Compromising the information within these nodes can severely impact an organization’s reputation and finances. Existing IDSs have struggled to identify various types of attacks, including zero-day attacks effectively, and also struggle with false alarms [6]. This has led to a demand for an improved NIDS that is efficient, accurate, and cost-effective, and can provide robust network security.

To create an effective IDS, researchers have used machine learning (ML) [7-9]. ML in AI extracts insights from huge datasets [10]. GPUs have helped these network security methods gain popularity in the past decade. Learning significant network traffic patterns using ML helps forecast normal and abnormal actions [11]. ML-based IDS uses feature engineering to analyze network traffic.

In Network Intrusion Detection Systems (NIDSs), supervised and unsupervised methods are used. The system is trained with normal or malicious network instances using labelled data in supervised techniques. The system recognizes similar cases after distinguishing these classes. However, unsupervised methods discover deviations in network traffic patterns. These techniques can detect intrusion-related behavior. Improve network security by combining supervised and unsupervised threat detection and response.

1.1 Problem statement

Network infrastructures need NIDS to prevent more complex and frequent intrusions [12]. Due to shifting data flows, traditional NIDS have high false-positive rates and difficulty identifying emerging attack vectors [13-15]. Supervised and unsupervised machine learning have been researched separately, but not together. This paper develops and tests a hybrid NIDS framework that combines supervised and unsupervised learning to increase accuracy, false positives, and intrusion detection in dynamic network environments.

1.2 Contributions

We revolutionize NIDS in this paper. Our hybrid architecture combining supervised and unsupervised machine learning is innovative. This blend combines supervised learning's precision, labeled dataset’s ability to identify and categorize known intrusion behaviors, and unsupervised learning's ability to exploit traffic abnormalities. Early discovery of these anomalies, which typically indicate new attack avenues, is critical for security.

Additionally, our system incorporates a dynamic algorithm tailored to autonomously modify its detection methodology in alignment with the characteristics of the incoming data stream. This ensures peak efficiency, even in continuously transforming network scenarios. Our comprehensive assessments, conducted using a variety of datasets, pit our pioneering model against conventional supervised and unsupervised methods. The outcomes, both in quantitative and qualitative terms, underscore the heightened detection capabilities, diminished false alarms, and increased accuracy rates achieved by our blended approach.

In conclusion, our research carves a pathway for academics and industry professionals aiming to craft a robust, flexible, and efficacious NIDS. Our solution addresses the pressing demands of today’s digital landscape, which is continually threatened by sophisticated cyber adversaries, by harnessing the power of anomaly exploitation for enhanced security.

1.3 Paper structure

The paper commences with an Abstract, providing a succinct overview of the study’s objectives, methodology, and primary findings. Following the abstract, the Introduction sets the stage, highlighting the topic's significance, outlining the primary research question, and providing context. Delving deeper, the Related Work section critically examines existing research pertinent to the study, identifying gaps and delineating the current study’s significance within the broader academic landscape. Subsequent to this, the Methodology section elucidates the research design, elaborating on data collection, preprocessing, and the analytical techniques employed. Once the foundational processes are established, the Results section presents the empirical findings derived from the analysis, often complemented by tables, charts, and other illustrative tools. To interpret these findings. The study then culminates in the Conclusion, summarizing the key and implications. Finally, Future Work offers directions for subsequent research endeavors, and the References section lists all the scholarly works cited throughout the paper, ensuring academic rigor and integrity.

In our methodological approach, we commence by sourcing our dataset from Kaggle. With the data at hand, an exploratory data analysis (EDA) is conducted, enabling us to delve deeply into the underlying patterns and characteristics of the data. Post-analysis, we embark on the preprocessing phase to refine the dataset and make it more conducive for subsequent operations. Our primary innovation lies in the hybrid techniques we employ: combining DBSCAN with RF and juxtaposing it with the integration of K-means with RF. These combinations aim to harness both the power of clustering (unsupervised learning) and classification (supervised learning) to enhance intrusion detection capabilities. Following the hybrid modeling, clustering techniques are applied to further segment and categorize the data. The culmination of our process is the evaluation phase, where the efficacy of our proposed methods is rigorously tested and benchmarked. Figure 1 succinctly encapsulates our methodology in a visual flowchart. Each of these stages will be elaborated upon in the subsequent subsections as shown in Figure 2.

1.png

Figure 1. The class distribution of a dataset

e06f0808-c0de-4ccb-ab00-cfdfedbfb04e.png

Figure 2. Proposed flowchart

3.1 Dataset

Our study utilizes a dataset derived from a sophisticated simulation that replicates a typical United States Air Force Local Area Network (LAN), aimed at capturing a diverse range of cyber-attacks within a realistic network environment. The study we conducted required authentic TCP/IP dump data, which this configuration provided. Each connection in the dataset is a sequence of TCP packets sent between a source and a target IP address over a defined duration using specific protocols. These connections are carefully classified as 'regular' or 'anomalous' and assigned an attack type.

The dataset is detailed, with 100 bytes per TCP/IP connection. It has 41 attributes—3 qualitative and 38 quantitative. Protocol type, service, and complex metrics like failed login attempts and service request rates are included.

It defines 13,449 cases as 'Normal' and 11,743 as 'Anomalous' across 22,544 rows and 41 columns. A complete network intrusion susceptibility analysis needs this balance. Evaluation of hybrid intrusion detection models is supported. The variables'duration', 'protocol_type','service', and 'flag' show the network's normal and compromised activities. Content diversity and balanced class distribution help understand network operations, making the dataset appropriate for intrusion detection research.

In Figure 1, 13,449 'Normal' network connections are '0' while 11,743 'Anomalous' connections are '1'.

3.2 Exploratory data analysis

Researchers need exploratory data analysis (EDA) to understand dataset complexity and patterns. This method finds anomalies, outliers, distributions, and assumptions using visual and quantitative data analysis.

EDA visualized "Normal" and "Anomalous" linkages for our dataset using simulated military networks. Balance—or potential imbalance—was obvious between these classes. Histograms, scatter plots, and box plots showed 41 qualitative and quantitative factors' distribution and links. The means, medians, standard deviations, and interquartile ranges of each feature were estimated. This highlighted the data's main trend and distribution.

We also created correlation matrix heatmaps to understand how features link, which could help with feature selection or engineering.

Visual cues and statistical testing detected anomalies and outliers that could bias our models. We performed a preliminary feature priority ranking to determine which qualities may be more important in selecting connection kinds.

In summary, the EDA step helped us grasp the dataset's features and guide preprocessing and modeling.

3.3 Preprocessing

The data analysis pipeline requires preprocessing to prepare and shape the dataset for modeling. In this phase, data quality is improved by several activities. We preprocessed our dataset as follows:

• Missing Data: Addressing missing data was the first step. Incomplete data can influence analyses. The dataset was checked for nulls and missing values. Depending on their nature and impact, these were either filled using mean, median, or mode or eliminated if they were negligible.
• Removal: Duplicate entries can skew results by giving repeated records undue weight. The dataset was thoroughly searched for and deleted duplicates. Each record is individual and contributes to analysis separately.
• Label encoding: For machine learning models, convert qualitative dataset attributes to numerical format. Used label encoding. A distinct integer for each qualitative feature value made them more modelable without compromising category meaning or relationship.

After these preprocessing steps, our dataset was clean and ready for research.

3.4 Dataset splitting

Dataset partitioning is necessary before model training and testing. This ensures that we have different data sets for training and testing our model. A solid split prevents overfitting and helps the model generalise to new data.

Recent research on dataset splitting ratios in machine learning algorithms supports a 70-30 data split for training and testing [1, 2]. Muraina [35] emphasized the importance of dataset splitting ratios in determining machine learning model parameters that best fit training data. Muraina [35] found that partitioning datasets into train and test sets is essential for model training and assessment after experimenting with 50:50, 60:40, 70:30, 80:20, and 90:10. Joseph [36] also recommended a splitting ratio of p: 1, where p is the number of parameters in a linear regression model that describes the data effectively.

Given this, we chose the typical 70-30 split ratio for our study.

• Training Set (70%): This 70% of the dataset is essential for creating and training machine learning models. By showing the model most of the data, we ensure it learns the dataset's patterns and correlations.
• Test Set (30%): The remaining 30% of the data is reserved for testing. This subset is crucial in gauging the effectiveness and accuracy of our models. Since the model hasn’t been exposed to this data during training, the test set offers a reliable measure of how the model will perform on real-world, unseen data.

It’s worth noting that while splitting, care was taken to ensure that both training and test sets are representative of the overall dataset, maintaining the inherent distribution of ‘Normal’ and ‘Anomalous’ connections. This is essential to ensure unbiased training and accurate performance evaluation.

3.5 Hybrid supervised and unsupervised techniques

Leveraging the strengths of both supervised and unsupervised learning paradigms, we introduced a hybrid approach aimed at enhancing the intrusion detection capabilities. This innovative methodology seeks to combine the structured, label-dependent learning of supervised techniques with the pattern discovery prowess of unsupervised ones. We merged RF with DBSCAN and K-means clustering methods in our study.

• RF + DBSCAN:
  • DBSCAN stands out for its ability to identify outliers and clusters of varied densities and shapes in datasets. This is useful for intrusion detection, because abnormal data points (possible intrusions or attacks) may not fit typical cluster shapes. DBSCAN clusters dense regions and labels sparse points as noise or outliers.
  • RF is a powerful ensemble learning algorithm used on clustered data. Cluster labels, added to the original data, let RF analyze and learn from data distribution and features within each cluster. Give RF more data structure context to boost classification performance and make more accurate and robust predictions.
• RF + K-means:
  • Data is clustered into a preset number of clusters using K-means. K-means divides the dataset into 'K' groups based on feature similarity, unlike DBSCAN, to uncover intrinsic data groupings. This approach organizes material into clear chunks well.
  • RF is used to classify data after K-means clustering, with the clusters serving as additional features. This lets RF use K-means' structured data to better comprehend inter-cluster interactions and improve the classifier's predictions.

We want to combine the strengths of supervised and unsupervised learning in our hybrid approach. The fusion enhances feature space and more. The enhanced feature set helps RF understand data structure and analyze more thoroughly by introducing cluster assignments. Clustering and classification enable the hybrid model discover threats in complex situations like intrusion detection. DBSCAN strengthens the model's tolerance to noise and outliers, helping it find data anomalies.

However, this integrated strategy has limitations. Clustering increases RF classification computing and training time. Also, changing K-means and DBSCAN parameters substantially impacts performance. Poor configuration can reduce clustering efficiency and model performance. Finally, cluster quality determines RF classifier success.

Poor clustering could confuse the RF algorithm, causing erroneous classifications. Despite these shortcomings, the hybrid model's ability to improve forecast accuracy and robustness, especially in complex and ambiguous domains, makes it valuable to intrusion detection.

3.6 Clustering

Clustering is a key unsupervised learning approach that organizes a dataset into clusters where instances in the same group are more similar than those in other groups. Clustering can help identify patterns, categorize intrusions, and identify new threats in intrusion detection.

Our strategy used two popular clustering algorithms:

• DBSCAN uses density instead of a preset number of clusters like many other clustering methods. This technique can discover clusters of different forms and densities from densely packed dots. Its unique capacity to differentiate outliers or noise can help detect aberrant actions that depart from patterns.
• K-means: A partitioning technique that groups data into 'K' different clusters based on feature similarities. The algorithm modifies cluster centroids until optimal partitioning is reached. K-means' deterministic nature helps it create separate, non-overlapping clusters for dataset categorization. The optimal 'K' typically requires additional procedures or heuristics like the Elbow approach.

Analyzing the clusters after clustering revealed the segmented groups' features. This step is vital to comprehend the variety of intrusions and the inherent patterns within the dataset. Moreover, the clusters serve as foundational inputs for subsequent modeling stages, especially in the hybrid supervised-unsupervised approach we’ve adopted.

3.7 Evaluation

Our IDS needs stringent evaluation criteria to prove its usefulness and dependability. These metrics reveal model performance and guide revisions and iterations. We used standard evaluation metrics in our research.

• The Confusion Matrix is a tabular form that details the model's predictions. Matrix includes:
• True Positives (TP): attacks case is correctly identified.
• True Negatives (TN): normal cases are correctly diagnosed.
• False Positives (FP): Misidentification of normal cases as attacks.
• False Negatives (FN): Misidentification of attack situations as normal.
• Accuracy: The percentage of correctly classified occurrences compared to the total instances.

$Accuracy =\frac{T P+T N}{T P+T N+F P+F N}$     (1)

• Precision: Assesses model accuracy, providing the percentage of detected attacks that were real.

$Precision =\frac{T P}{T P+F P}$     (2)

• Recall (Sensitivity): It evaluates the model's capacity to detect all attacks.

$Recall =\frac{T P}{T P+F N}$     (3)

• The harmonic means of Precision and Recall, F1-Score, balances their divergent scores.

$F 1-score =2 \times \frac{Precision  \times Recall }{ Precision +  Recall }$     (4)

• Specificity assesses the model's ability to accurately identify normal occurrences, indicating its genuine negative rate.

$Recall =\frac{T N}{T N+F P}$     (5)

• ROC Curve: A graphical figure demonstrating the diagnostic effectiveness of a binary classifier system with varying discrimination thresholds. The curve shows a true positive rate (Recall) against a false positive rate (1 - Specificity), and the Area Under the Curve (AUC) measures performance regardless of the categorization threshold.

These metrics collectively offer a holistic view of the model’s performance, ensuring it not only identifies attacks with precision but also minimizes false alarms. They are paramount in upholding the trustworthiness and operational reliability of the IDS.

The advent of hybrid methodologies, which amalgamate both supervised and unsupervised approaches, signifies a paradigm shift in intrusion detection systems. The synthesis offers the robustness of supervised models with the versatility of unsupervised techniques, ideally balancing precision with adaptability.

Referring to Table 5, the presented models’ performance across various research efforts is summarized. It becomes evident that while individual models, like Stacking or Random Forest, achieve commendable accuracy, none quite reach the performance pinnacle of the hybrid RF+DBSCAN approach proposed in this research.

Specifically, as illustrated in Table 7 the closest competing model is the stacking approach from the reference [37], registering an accuracy of 97.3%. However, even this impressive figure falls short of the 99.70% achieved by the RF+DBSCAN combination. It’s intriguing to note that the Long Short-term Memory combined with Extreme Boosting, as described by Thapa et al. [21], did not perform as efficiently, which underscores the point that not all hybrid models guarantee top-tier performance. Nonetheless, the superior efficacy of our proposed model reaffirms the potential of judiciously integrating supervised and unsupervised techniques.

Another observation to highlight is the performance of the Random Forest model from Song et al. [22], which achieves an accuracy of 97.01%. This suggests that the Random Forest algorithm, even when used in isolation, exhibits robust performance. Yet, the fusion with DBSCAN in our proposed approach amplifies its potency, pushing the accuracy to a near-perfect score.

The Generative Adversarial Network [26] and the Hierarchical Temporal Memory [27], while employing distinct methodologies, converge around the same accuracy range, further accentuating the varied outcomes different models can achieve when addressing similar problems.

In essence, the results accentuate a pivotal inference: while individual models can achieve remarkable accuracy, their performance can often be enhanced through hybrid methodologies. The synergy of RF+DBSCAN, as proposed, champions this idea, combining the precision of RF with the clustering prowess of DBSCAN. The data shows that this confluence creates an optimal IDS that outperforms its components and several contemporaneous models.

Though powerful in detection, the RF+DBSCAN model requires more computational effort due to clustering and classification. Due to its complexity, the DBSCAN algorithm may take too long to locate clusters for real-time intrusion detection. This hybrid model's efficiency and efficacy depend on DBSCAN's parameter calibration, which if not tuned, could damage it.

RF+DBSCAN's near-perfect accuracy is obvious, but understanding and addressing its computational constraints will be crucial for its real-time adoption. This involves a careful balance between the depth of detection and the necessity for speed, ensuring that the model remains both accurate and agile in live network environments.

Table 7. Comparative analysis with state-of-the-art models