Intrusion Detection and Identification System Design and Performance Evaluation for Industrial SCADA Networks

This section presents the Gas Pipeline dataset [12], its features and attack class labels. It also presents the preprocessing steps and methods applied to the datasets to fill in the missing values.

2.1 Dataset features

The original dataset has 17 features and 3 different class label groups namely binary, categorized and specified. There are a total 274,628 instances in the dataset. There are 11 Command Payload features which are related to the command injection attacks, 5 Network features, and 1 Response Payload feature related to the response injection attacks as listed in Table 2. We next provide a brief description for each of the twenty features in Table 2. The detailed description of features and the associated collection method can be found in Turnipseed [18].

The station address feature is a unique eight-bit value assigned to each master and slave device. In broadcast mode, all slaves receive the transmitted frame and need to check the address field to determine if it is the intended recipient. This feature is useful for the detection of scan attacks. Up to 256 different function codes such as read and write commands can be executed in the system and this information is contained by the second feature. One typical attack leveraging this feature is the denial of service by forcing a slave to the listen-only mode. Modbus frame length which is fixed for command and response queries is contained by the third feature. A frame with a different length can easily be detected as not normal. Another feature indicates the set point value for controlling the pressure in the pipeline in automatic mode. Manipulation of this value by an attacker could cause major damage to the system. PID controller values such as gain, reset rate, dead band, cycle time, and rate are represented by five other features. The system’s duty cycle with three possible values is controlled and represented by another feature. System control through the pump or the solenoid is accomplished by the so-called control scheme which is contained by the eleventh feature. In the event the system mode is manual, the state of the pump, namely either off or on, is controlled by a dedicated field in the frame constituting the twelfth feature. The state of solenoid valve as either open or closed is controlled a dedicated field in the frame: the thirteenth feature contains this information. Tampering with the state of either the pump or the solenoid could cause serious damage. The current pressure measurement for the pipeline is contained by the fourteenth feature. The cyclic redundancy check data which facilitates checking for errors in a frame is contained by the fifteenth feature. An additional feature is included to distinguish between commands and responses. The last four features specify the time stamp, attack type, attack class, and attack vs. normal data.

2.2 Description of attacks

The gas pipeline dataset used in this study has 7 types or categories of attacks as presented in Table 3. The description for all attack types is given in Morris and Gao [12]. Naïve Malicious Response Injection (NMRI) and Complex Malicious Response Injection (CMRI) are the response injection attacks. These attacks can hide by mimicking certain behaviors which occur within normal operating bounds. This makes them very difficult to detect, and hence giving the appearance of the system operating normally. NMRI has out of bounds behavior that would not be present in normal operation. It typically occurs when the attacker lacks information about the physical system process. CMRI attacks provide a level of sophistication over NMRI attacks. These attacks can change the state of a system which can be seen as command injection attacks: they are difficult to detect.

Table 2. Original features in gas pipeline dataset [18]

Table 3. Attack classes in gas pipeline dataset

Malicious State Command Injection (MSCI), Malicious Parameter Command Injection (MPCI), and Malicious Function Code Injection (MFCI) labels belong to the command injection attacks. Tables 4 and 5 show their specific attack types and their adverse impact on the system. Much damage may originate from command injections attacks: interruption in device communications, modification of device configuration, and modification of the PID values are some of them. MSCI attacks modify the state of the current physical process of the system and can potentially place the system into a critical state. Table 4 specifies the MPCI attack types. It mainly modifies the parameters of PID configurations and set point. As listed in Table 5, MFCI attacks inject commands which exploit the network protocol for restarting, cleaning registers etc.

Table 4. MPCI attack subtypes

Table 5. MSCI, MFCI, DoS, recon attack subtypes

Denial of service (DoS) attacks are very common in almost every networked and online system. In a SCADA system, a DoS attack attempts to disrupt communication between the control or monitoring system and the process. Another category of attacks are reconnaissance attacks. These attacks aim to collect information about the system through some passive activity. They may also query the device for information such as function codes, model numbers etc. Specific attack types belonging to NMRI or CMRI are listed in Table 6.

2.3 Preprocessing

In the preprocessing stage, missing values in the dataset were filled in. There were missing values in the dataset for 11 Payload (10 Command Payload and 1 Response Payload) features. The missing values could have been imputed in multiple ways. For instance, Perez et al. [14] have imputed the missing values in 4 different ways such as mean value, keeping previous value, zero imputation, and K-means imputation for the same dataset. In the gas pipeline dataset, missing values were occurring as MAR (Missing at Random) or NMAR (Not Missing at Random). The payload features were not occurring at random as the solenoid or pump mode values were fixed among 0,1 or on/off/automatic. But the pressure measurement value was occurring randomly while an associated attack was in progress. Accordingly, the missing values were imputed with Multivariate Imputation by Chained Equation (MICE) method [29] since the MICE algorithm can handle both MAR and NMAR types. This type of imputation works by filling the missing data multiple times. Multiple Imputations are much better than a single imputation as it measures the uncertainty of the missing values more precisely [29]. The chained equations approach is also very flexible and can handle different variables or different data types.

Table 6. NMRI & CMRI attack subtypes

2.4 Dataset partitioning for training and testing

To create the train-test data subsets, we partitioned the original dataset into three splits and then formed training-testing datasets at a ratio of 66.6% and 33.3% and repeated three times to form three data folds. The original dataset was split into three equal parts while preserving class representations equally in each split. Each fold contains two splits for training and the third split is used for testing: this procedure generates a unique split for testing for a given fold.

Simulation study was performed to determine the performance of the 3-stage classifier on each of the three folds. Performance assessment and evaluation was done separately for each test data split and further inferences were made based on these three performance assessments and evaluations.

Training the 3-stage classifier was accomplished by implementing the following procedure:

Step 1: Train the binary classifier in Stage 1 with two class labels of Normal versus Attack.

Step 2: Remove all patterns with class label “Normal” and replace the single “Attack” label with 7 attack class labels in the training dataset.

Step 3: Train the 7-category classifier in Stage 2 with training dataset modified as in step 2.

Step 4: Split the single training dataset as modified in step 2 into 7 subsets (one for each attack class); replace, in each training subset, class labels with the corresponding attack type labels. A training data subset will have those records or instances belonging to specific attack types of a given attack class.

Step 5: Train each of the 7 classifiers in Stage 3 using corresponding training data subsets modified in step 4.

Performance assessment and evaluation was done by implementing the following procedure:

Step 1: Classify the pattern in the testing data subset with binary class labels using Stage 1 classifier.

If classification output is “Normal” then no further processing is needed.

Else (classification must be “Attack”) continue with Stage 2 processing.

Step 2: Classify the attack pattern into one of seven classes in the testing data subset using the 7-category classifier in stage 2. Based on the output from the single 7-category classifier, chose the Stage 3 classifier among 7 to activate.

Step 3: Classify the attack pattern in the testing data subset using the attack type classifier (among those 7 at stage 3) as identified in Stage 2.

4.1 Simulation results and discussion

Weka classifier Random Forest was used for all classifier models in all 3 stages [31]. The training set of the first data fold contains dataset splits 1 and 2 while the dataset split 3 is employed as the test set. Table 7 presents the number of instances available in both training and testing data subsets for stage 1. We report the representative results for one of the three data folds since results for all three folds are very similar to each other with negligible differences.

1a.png

(a) Stage 1 classifier

1b.png

(b) Stage 2 classifier

1c.png

(c) Stage 3 classifier

Figure 1. 3-stage classifier design

Table 7. Instance counts for stage 1

For stage 1, the batch size is 1000 for the Random Forest model which classified 89,906 instances correctly from the test set yielding 98.16% accuracy. The confusion matrix in Table 8 shows 18,597 attack instances were classified correctly among the 19,910 and thus resulting in 1,313 incorrectly classified attack instances. From Table 9, we see that false negative rate (FNR) for the attack class is 6.6% which corresponds to the 1,313 incorrectly classified attack instances.

Table 8. Stage 1 confusion matrix

Table 9. Stage 1 performance (accuracy = 98.16%)

Next, the training set is modified: all 142,901 normal class patterns are removed from the training set of stage 1. This leaves only the attack class instances in the training set for stage 2. Table 10 presents the number of instances available in training and testing data subsets for stage 2.

Table 10. Instance counts for stage 2

The batch size used for stage 2 classification is 100 as the number of instances decreased in comparison to stage 1. Random Forest at Stage 2 classifies with 93.79% accuracy where 18,674 test instances are correctly classified and 1,236 are incorrectly classified. The weighted FNR is 6.2% for stage 2. From the confusion matrix in Table 11 we see that the model struggles distinguishing mainly between attack classes 1 and 2. There are 1161 incorrectly classified instances between attack classes 1 and 2 which amounts to approximately 94% of incorrectly classified instances. NMRI and CMRI, being both response injection attacks, deal with mainly the pressure measurement features which have overlapping values for both attack classes 1 and 2. For example, both attacks have same pressure values in the range of 2 to 10 kPa. That is the most likely reason why the classifier struggles to distinguish between them. This finding also indicates the need to determine and formulate new features which would be discriminatory between attack classes 1 and 2.

Performance metric values for the stage 2 classifier are presented in Table 12. The accuracy is 93.79% for the classifier. Lower values for all the metrics for classes 1 and 2 reinforce the findings drawn from the confusion matrix that the design needs to be enhanced to be able to better distinguish between attack classes 1 and 2.

Table 11. Stage 2 confusion matrix

In stage 3, seven classifiers are used. Training and testing datasets of stage 2 are now divided into 7 separate training and testing subsets. Each of the 7 training and testing data subset pairs have associated subclass labels, which range from 1 to 35, belonging to corresponding stage 2 class labels. Table 13 presents the number of instances available in both training and testing data subsets for stage 3 classifiers.

Table 12. Stage 2 performance (accuracy = 93.79%)

In stage 3, the batch size is 10 for the Random Forest classifier as the test data subset size is further reduced following the processing as a result of classification during stage 2. Since the attack class 6 (DoS) has only 1 specified subclass (18) in the dataset, it was not necessary to build a classifier for it. Tables 14 through 19 present the confusion matrices for all 6 other subclass classifiers.

Table 13. Instance counts for stage 3

For class label 1, the confusion matrix is presented in Table 14: the classifier struggles to distinguish between attack subclasses 31 and 32. Attack subclass 31 is dependent on the random pressure measurements sent to the device and attack subclass 32 is dependent on the negative pressure readings sent to the device. The classifier fails to detect these two attacks whenever the random pressure sent to the device is also negative and detects 31 as 32 or 32 as 31. Additionally, subclass 29 also has relatively poor detection rates as it is misclassified as subclass 31 or 32 while a good number of patterns belonging to attack subclasses 31 or 32 are also classified as 29. The classifier for attack class 1 achieves an accuracy rate of only 82.30%. The FNR is 17.7%, which is very high compared to other classifiers for the same metric. Performance of classifier for attack class 2, presented in Table 15, is the second worst with 87.79% accuracy and 524 incorrectly classified instances. All other classifiers performed at a much higher level, specifically 99% or better as shown in Tables 16 through 20.

Table 14. Stage 3 class label 1 confusion matrix

Table 15. Stage 3 class label 2 confusion matrix

Table 16. Stage 3 class label 3 confusion matrix

4.2 Comparison with studies reported in literature

Table 17. Stage 3 class label 4 confusion matrix

Numerous studies using the same gas pipeline dataset were reported in the literature: several of them only reported the binary classification results [22-27] while others considered the classification problem for seven attack classes as presented in Table 21 [14-16, 19-21]. Many appear to employ at least one aspect of data preprocessing, classifier design and testing that could render the comparison with the design proposed in this study not very meaningful or even valid. Nevertheless, for the sake of establishing somewhat meaningful context for the performance of the proposed design, we briefly discuss each one next. For a comparison with the other studies in the literature on the same dataset, it is then necessary to calculate the combined performance of stages 1 and 2 for the design proposed in this study. The accuracy of combined stages 1 and 2 is 0.9379×0.9806= 92.06%. Precision and recall rates are 0.937×0.982=0.920 and 0.938×0.982=0.921, respectively.

Table 18. Stage 3 class label 5 confusion matrix

Table 19. Stage 3 class label 7 confusion matrix

Table 20. Stage 3 performance

Demertzis et al. [16] and Nazir et al. [21] proposed one-class anomaly detection approach where the former as also reported subsampling the dataset. Perez et al. [14] reports better performance compared with that of the current study, and yet their approach employed 80-20% ratio for splitting the dataset into training and testing subsets. Results reported in the studies [15, 19] suggest poor performance across all the metrics considered.

Table 21. Performance comparison of 2-stage classifier with studies in literature

The only other cited study which attempted to address the classification problem for the case of 35 attack types through a 3-stage design reported precision, recall and f-measure values of 0.586, 0.434, and 0.445, respectively [24]. Comparing these performance metric values with those in Table 20 shows that the 3-stage design proposed in our study performed very well.