A Hybrid Intrusion Detection System with Quantum Epigenetic Optimization and Autoencoder Feature Reduction

Digital threats represent one of the most prominent challenges facing nations in the modern era, as electronic systems have become a fundamental component of economic infrastructure. In Iraq, the digital economy is gaining increasing importance in supporting and developing various sectors, such as industry, services, and trade. However, the country faces escalating threats from cyberattacks targeting protected information, whether it is personal data or data belonging to institutions or other groups. This includes sensitive data such as financial system information and data related to assets and the daily operations of companies and institutions, directly threatening the stability of the national economy [1, 2].

In order to identify and stop hostile activity in real time, an intrusion prevention system (IPS) continually analyses network as well as system traffic. By automatically stopping assaults before they cause harm, in addition to recognizing threats, it expands the capabilities of classical IDS, promptly detects and stops harmful traffic, maintains a constant watch on host and network activity, uses abnormality analysis, behaviour, and indicators, and applies automatic responses to stop incursions, which increases the visibility of security and lessens the effect of attacks [3, 4].

Even with the tremendous advancements in security detection technologies, a majority of those in use today still have basic problems. These include the need for individual feature selection, the large size of the information, which increases the level of computation as well as the number of false alarms, and the difficulties in attaining almost immediate effectiveness [5, 6].

Most IDSs are machine learning-based and use hand-chosen features, which can result in redundancy, overfitting, and higher computational costs [7, 8]. Also, deep learning models may have a high ability to capture complex patterns, but they normally require sequential data preparation and time-consuming training; thus, it is difficult to implement them on a real-time basis. Additionally, evolutionary optimization [9] with deep learning hybrid approaches has not been intensively studied on large benchmark datasets, and therefore, a research gap exists in scaling and resource-efficient IDS systems. There is also a lack of systematic comparison between conventional and deep learning approaches, especially when feature compression methods are used [10, 11].

Another important gap is in the assessment of the metrics and the reliability of IDS models with different types of attacks. Most research documents indicate that a classifier can be highly accurate on a particular set of attacks but cannot be generalized to handle different sets of threats [12, 13]. Besides, the effect of feature compression on model interpretability and detection speed is not well studied. The following gaps reveal why it is important to have a holistic framework that incorporates evolutionary feature optimization, dimensionality reduction, and conventional and deep learning classifiers to deliver robust, scalable, and near real-time IDS performance [14].

Quantum computing is a paradigm shift in the way data is processed, and it is able to do things that classical computers cannot, because of the characteristics of superposition and entanglement, which can process large volumes of data at once [14]. Quantum computing is becoming a crucial component of intrusion detection systems (IDS) development in the area of cybersecurity because these systems are experiencing considerable pressure due to the sheer rise in the amount of data, the complexity of cyberattacks, and the sophistication behind the current types of intrusions. It is no longer enough to use classical computing to guarantee the detection of complex attacks or the reduction of false positive rates; therefore, the so-called hybrid Quantum-Classical IDS, where quantum computing can be used to process high-dimensional data and classical computing is used to perform regular tasks, has appeared [14, 15].

It is now possible to produce more accurate, sensitive, and false-positive suspicious activity detection, including large-scale, multi-layered attacks, with Quantum Machine Learning (QML) algorithms like Quantum Neural Networks (QNNs) and Quantum Support Vector Machines (QSVMs). In addition, quantum computing strengthens network security through Quantum Cryptography, such as Quantum Key Distribution (QKD), which ensures security against future attacks, including those that may be based on quantum computing. Also, quantum algorithms like Grover's Search can scan millions of possible scenarios of network activities in a short time, significantly reducing response time and increasing the system's ability to identify complex malicious behaviors that a classical system may fail to detect [14, 16]. Overall, quantum computing is seen as an enabling technology for the future of cybersecurity, offering advanced solutions to enhance the effectiveness and accuracy of intrusion detection systems and opening possibilities for designing smarter and more flexible systems capable of responding to constantly changing cyber threats, making it a major part of modern information security policies [14, 17].

1.1 Problem statement

High efficiency, computing economy, and immediate performance are challenges faced by modern IDS. Deep neural network algorithms are expensive and require lengthy training periods, while individually selecting characteristics leads to redundant or unimportant characteristics. Although techniques like quantum autoencoder and Quantum Epigenetic Algorithm (QEA) have been successful in raising precision and decreasing false positives, there are few studies on the detection of anomalies using quantum computing [18-21]. To achieve quick, precise, and sustainable detection, blended frameworks combining deep learning and quantum computing must be developed.

1.2 Purpose of the study

This research aims to create a strong, scalable, and resource-saving IDS structure by:

• Features subset optimization using a Quantum Epigenetic Algorithm (QEA);

• Incorporating feature compression (Autoencoders) to compress the data;

• Comparison and evaluation of the performance of the classical classifiers (Random Forest, Gradient Boosting) and deep learning architectures (DNN, LSTM) on the UNSW-NB15 benchmark dataset;

• Providing evidence that hybrid optimization and representation learning can enhance detection accuracy, reduce false positives, and still perform in near real time.

The rest of this paper is structured as follows: Section 2 presents related works. The methodology is provided in Section 3. Section 4 presents results and discussion, with comparative analysis. Section 5 provides the conclusion, summarizing findings, contributions, limitations, and future research recommendations.

The adopted research design in this study is hybrid in nature and combines evolutionary optimization with the traditional and deep learning classifiers to compare the performance of intrusion detection. Its design includes consecutive phases of data preprocessing, feature selection, feature compression, and classification so that it could be reproducible and scalable. It takes advantage of the UNSW-NB15 data benchmark where a normal, as well as attack network traffic, is provided. The study design is quantitative, in which the measures of predictive performance including accuracy, the area under the curve, and little computational time are measured to compare the results. Figure 1 shows the phases of the hybrid experimental research design that will be used in this study.

Figure 1. Workflow of proposed system diagram

The process will start with loading the UNSW-NB15 dataset and then data preprocessing which involves encoding categorical features and normalization of numeric features. In the case of feature compression, an autoencoder will compress the dimensions of the input features; in this situation, the raw features are used. Quantum Epigenetic Algorithm (QEA) is subsequently used to perform the process of feature selection iteratively by incorporating the aspect of fitness evaluation, selection of the most performing individuals and generation of offspring in the course of several generations. QEA is a quantum-inspired classical algorithm and does not require actual quantum hardware. After selecting the optimal set of features, several different classifiers are trained and tested, such as Random Forest (RF), Gradient Boosting (GB), Deep Neural Networks (DNN), as well as Long Short-Term Memory (LSTM) networks. This pipeline guarantees the optimization of features and classifier validation systematically, intending to maximize detection performance, minimize false positives and preserve computational efficiency.

3.1 Sampling method

The data will consist of 175,341 training samples and 82,332 test samples, with 49 raw features and a class label of each. The stratified sampling approach was used to maintain the ratio of attack and normal classes in training and testing sets and reduce the effects of class imbalance. This method guarantees that the training and evaluation stages have a similar proportion of known and novel types of attack, enhancing generalization [3, 4]. The sampling can also be used to cross-validate in the process of evaluating the feature selection and classification methods and can be used to make statistically significant comparisons.

3.2 Data collection techniques

Data was collected from the UNSW-NB15 dataset, which aggregates synthetic and real network traffic, simulating modern cyber-attack scenarios. The raw dataset includes numeric and categorical features, requiring preprocessing steps such as one-hot encoding for categorical attributes and normalization for numerical attributes. Preprocessing was conducted as follows:

• Categorical encoding: All nominal features were transformed using one-hot encoding to convert them into numerical format suitable for machine learning algorithms.

• Normalization: Features were scaled between 0 and 1 using min-max normalization to facilitate neural network convergence.

• Feature compression: An autoencoder-based architecture was trained to reduce dimensionality while preserving critical information, resulting in a latent feature space of 64 dimensions.

3.3 Data analysis methods

The analysis will use three key steps as follows: (1) feature optimization, (2) model training and (3) evaluation. An algorithm called Quantum Epigenetic Algorithm (QEA) was used to identify the best subsets of features by maximizing a fitness function comprising accuracy and AUC. The fitness is determined as follows:

$Fitness =0.7 \times Accuracy +0.3 \times A U C$          (1)

RF, DNN and LSTM classifiers were then trained using selected feature subsets. In the case of LSTM, data was sequentially preprocessed using each sample as a temporal sequence of features. RF and Gradient Boosting (GB) classifiers were used to assess the Autoencoder-compressed information. The accuracy, Area Under the ROC Curve (AUC), and confusion matrices were used to evaluate model performance. Eq. (2) represents the measure of accuracy:

$Accuracy =T P+T N / T P+T N+F P+F N$          (2)

and Eq. (3) represents AUC measure:

$A U C=\int_1^0 \frac{1}{T P R(F P R(x))} d x$          (3)

where, TP, TN, FP, and FN denote true positives, true negatives, false positives, and false negatives, respectively.

The QEFO Algorithm illustrates as:

3.4 Mathematical formulation of QEA

a) Modeling quantum bits

Each solution (feature subset) is represented as a vector of qubits:

$Q=\left\lceil\left[\alpha_1, \beta_1\right],\left[\alpha_2, \beta_2\right], \ldots,\left\lceil\alpha_n, \beta_n\right\rceil\right\rceil$          (4)

where,

$\left|\alpha_{\mathrm{i}}\right|^2+\left|\beta_{\mathrm{i}}\right|^2=1$          (5)

αᵢ: Probability that the feature is active (1).

βᵢ: Probability that the feature is inactive (0).

b) Observation (feature selection)

During observation, a binary vector is generated as follows:

$x_i={1\ if\ {rand}(0,1)<\left|\alpha_{\mathrm{i}}\right|^2; 0\ otherwise}$        (6)

c) Epigenetic regulation (activation function)

A dynamic regulation layer inspired by epigenetics modifies the feature activation:

$g_{\mathrm{i}}=x_{\mathrm{i}} \cdot \sigma\left(w_{\mathrm{i}}\right)$          (7)

$\sigma\left(w_{\mathrm{i}}\right)=1 /\left(1+e^{\wedge}\left(-w_{\mathrm{i}}\right)\right)$          (8)

where,

gᵢ is the final state of the feature.

wᵢ is a dynamic weight.

d) Fitness function (multi-objective)

The fitness of each candidate solution is evaluated using a multi-objective function:

$\left.F=\lambda_1 \cdot A C C-\lambda_2 \cdot F P R-\lambda_3 \cdot(|S| / N)-\lambda_4 \cdot L A\right)$          (9)

where,

ACC: Classification accuracy

FPR: False positive rate

|S|: Number of selected features

N: Total number of features

LAT: Inference latency

λ₁, λ₂, λ₃, λ₄: Weight coefficients

e) Update rule for qubits

After evaluation, qubits are updated using a rotation gate:

$\begin{aligned} & {\left[\alpha_{\mathrm{i}}{ }^{\prime}\right][\cos (\Delta \theta)-\sin (\Delta \theta)]\left[\alpha_{\mathrm{i}}\right]} \\ & {\left[\beta_{\mathrm{i}}{ }^{\prime}\right]=[\sin (\Delta \theta) \cos (\Delta \theta)]\left[\beta_{\mathrm{i}}\right]}\end{aligned}$          (10)

where,

Δθ is the rotation angle depending on the fitness of the solution.

3.5 Model hyperparameters

As a measure to generate reproducibility and compare the models fairly, Table 2 presents the key hyperparameters involved in training the models. These parameters were chosen according to experimental initial work and the best practices in literature.

Table 2. Model hyperparameters

Compression with autoencoders to 64 latent dimensions was effective in reducing computational cost and at the same time had competitive accuracy (RF: 83.65, GB: 85.50) although at the cost of a small AUC. Quantum Epigenetic Algorithm (QEA) had the highest performance with RF + QEA having a high accuracy of 87.08 and an AUC of 0.9739 with RF + QEA having a high trade-off between accuracy and efficiency. DNN (87.0%, AUC 0.9735) and LSTM (86.5%, AUC 0.972) were also doing well, although it took LSTM more time to process and train.

Despite elucidating the fundamental concepts, outcomes mostly concentrate on performance indicators like accuracy and AUC.  The key to QEA's efficacy is its capacity to identify a variety of important traits, which lowers noise and false positives while increasing accuracy.  Performance reduction is negligible because autoencoder-based feature compression maintains the key patterns with relatively little data loss.  This analysis explains the framework's success and viability for nearly immediate deployment by demonstrating how the mixed method (QEA + autoencoder + deep learning classifiers) improves identifying attacks and decreases time to react. In general, the QEA-based feature selection method is more stable and has fewer false positives than conventional baseline methods, whereas the autoencoder compression method makes inference in resource-constrained settings nearly real-time. These results point to hybrid IDS models as efficient, scalable, and effective in countering emerging cyber threats.