Payload-Oriented Intrusion Detection Through Packet-Flow Association and Cryptography-Driven Statistical Features

The rapid proliferation of networked environments, combined with the increasing sophistication of cyberattacks, has made intrusion detection systems a vital component of modern cybersecurity infrastructures [1]. Over the past few years, a lot of focus has been directed towards the machine learning-based intrusion detection systems because of their ability to identify known and unknown cyberattacks [2-4]. Most of the existing studies are mainly based on features extracted from the flow level of datasets, in which the analysis is mainly limited to statistical characteristics such as packet count, flow duration, and inter-arrival times, among others [5, 6]. Although these features are very useful in many intrusion detection situations, based on existing research, the statistical properties of packets inherent to their payloads have not been extensively studied. To this end, emerging research has started to investigate the use of payload-level statistical features in anomaly detection; for example, it has been suggested that entropy-based measures have the ability to provide unique information about the normal and anomalous traffic patterns [7]. In other words, entropy is a good measure of payload randomness [8], and hence it can be used to detect encrypted, compressed, or obfuscated traffic.

Inspired by these findings, this study further extends payload-driven analysis by incorporating chi-square deviation [9] and bit balance [7, 9, 10] as integral cryptography-driven features. The chi-square statistic measures the deviations from the payload byte-frequency distributions; bit balance is used to measure the deviations from the structural balance of the payload binary values. These features can be used together to provide a more comprehensive representation of traffic than flow statistics alone.

The ability to extract such features, however, requires direct access to raw packet capture traffic and not just relying solely on pre-computed comma-separated values (CSV) datasets. Indeed, there are a number of tools for accessing the contents and other information of flows [11, 12] but the proposed framework is not limited to packet extraction. Rather, it is the problem of building a scalable framework of associations between packets and flows, optimized for payload-based intrusion detection. Therefore, this study introduces a temporal hashing technique to reduce the amount of redundant comparison processes and to enable direct aggregation of statistical payload features within a machine learning-oriented intrusion detection system pipeline.

Consequently, the proposed framework makes contributions at multiple levels. First, a temporal hash-based indexing strategy is introduced for associating packets with flows. Second, it enriches conventional flow feature representation by directly extracting entropy, chi-square deviation, and bit balance from packet payloads. Third, a deterministic packet-to-flow assignment method ensures that each packet is linked to no more than one suitable flow, resolving ambiguities from temporally coinciding flows and prohibiting the multiple allocation of the same packet. Fourth, a session-aware data partitioning approach is introduced, in which related bidirectional flows are grouped into communication sessions prior to the establishment of training, validation, and test sets, thus eliminating the duplication of sessions across data partitions and lessening the likelihood of data leakage. Finally, it assesses the engineered features utilizing different supervised learning models and Shapley Additive Explanations (SHAP) [13] interpretable analysis. The integrated contributions allow a more discriminating and explainable intrusion detection system that is able to include statistical features which are usually neglected in conventional intrusion detection systems.

The remainder of this study is structured as follows. Section 2 critically reviews the relevant literature. Section 3 introduces the proposed framework and details the following stages: flow construction, packet-to-flow association, payload feature derivation, classification, and explainability evaluation. Experimental conditions are described in Section 4 and the results are presented in Section 5. Finally, Section 6 concludes the study and outlines directions for future research.

This study proposes an improved traffic representation framework to improve intrusion detection performance within supervised learning settings. This framework consolidates flow-level and packet-level information via a well-organized mechanism for packet-to-flow matching, while incorporating payload-based statistical feature extraction. In contrast to traditional flow-oriented methods that depend entirely on aggregated statistical features, the proposed framework augments flow records by per-packet information, providing a richer representation of network traffic behavior, as illustrated schematically in Figure 1.

The proposed framework consists of five main stages:

Stage 1: Flow construction from original packet capture traffic.

Stage 2: Effective packet-to-flow association based on temporal ordering indexing.

Stage 3: Feature augmentation utilizing payload-inferred statistics in parallel with the traditional ones.

Stage 4: Machine learning-based classification for anomaly detection.

Stage 5: Evaluation of model interpretability using SHAP.

1.png

Figure 1. Pipeline of the proposed payload-based intrusion detection framework

The designed framework enables scalable processing of large-scale network traffic captures while ensuring temporal consistency between packets and flows. Although tools, such as Wireshark [11] and TShark [12], enable extensive packet inspection and protocol decoding functionalities, this framework goes beyond packet representation and field derivation. Instead, it deals with effective packet-to-flow association and payload-derived statistical aggregation for forming consolidated flow-based feature representations. The following subsections describe each component in detail.

3.1 Flow construction stage

This stage reconstructs network flows straight from unprocessed packet capture traces. This stage is essential, as all successive stages depend on a rigorous and temporally coherent representation of network flows.

A naïve approach to performing this stage would involve comparing every incoming packet with all present flows. However, if the number of packets is $P$ and the number of flows is $N$, then a brute-force solution would incur a considerable time complexity represented by $O(P \times N)$. Such computational complexity is impractical for addressing large-scale network traces, particularly when processing millions of packets. To cope with this limitation, this study utilizes a hash-indexed strategy whereby flows are grouped and indexed in accordance with their network keys and session interval. As a result, the search operation is narrowed from scanning the entire flow table to only a limited subset of candidate flows sharing the same key and falling within the same time window. The following formal definitions and algorithms describe the proposed solution in detail.

Definition 1. Let the set of network flows be denoted by $\mathcal{F}=\left\{\mathcal{F}_1, \mathcal{F}_2, \ldots ., \mathcal{F}_N\right\}$, where each flow $\mathcal{F}_i$ with index $i$ is identified by a 3-tuple, $\left(k_i, t_i^{\text {start}}, t_i^{\text {end}}\right)$. Here, $k_i=\left(\right.$srcIp$_i$, dstIp$_i$, srcPort$_i$, dstPort$_i$, proto$\left._i\right)$ represents the standard 5-tuple key, which consists of the following session components pertinent to $\mathrm{F}_i$: source and destination of Internet Protocol (IP) addresses, source and destination port numbers, and the protocol identifier. Furthermore, $t_i^{\text {start}}$ and $t_i^{\text {end}}$ represent the start and end times of flow $\mathcal{F}_i$, respectively.

Definition 2. To efficiently organize the flows, the temporal flow index is specified as a hash structure $\mathcal{H}$. This structure is formulated as a mapping function $\mathcal{H}: \mathrm{K} \rightarrow \mathcal{L}$, such that K is the set of all viable network keys, and $\mathcal{L}=\bigcup_{k \in K} L_k$, such that $L_k$ is the list of 3-tuple of the form $\left(t_i^{\text {start}}, t_i^{\text {end}}, i\right)$. Hence, $L_k$ contains all flows associated with the key $k_i \in \mathrm{~K}$. In order to enable effective temporal matching, this list is arranged in non-decreasing order in regard to the flow start times, i.e., $t_i^{\text {start}} \leq t_j^{\text {start}}$ for any two members with indices $i<j$, it holds that $\forall i, j \in\left\{1, \ldots,\left|L_k\right|\right\}, i<j \Rightarrow t_i^{\text {start}} \leq t_j^{\text {start}}$.

Definition 3. For each key $k_i \in \mathrm{~K}$, a temporal indicator is defined as $\mathfrak{D}$ which denotes the current scanning position within the flow list $L_k$. The purpose is to eliminate unnecessary rescanning of inactive flows and enabling effective packet-to-flow association under large-scale traffic streams.

Definition 4. Assume that the packet stream is expressed as: $p=\left\{p_1, p_2, \ldots, p_P\right\}$, and every packet $p_j$ with index $j$ can be formulated as a 2-tuple: $\left(k p_j, t p_j\right)$, where $k p_j$ and $t p_j$ are the key and the timestamp of packet $p_j$, respectively, and $k p_j=$$\left(\right.$srcIp$_j$, dstIp$_j$, srcPort$_j$, dstPort$_j$, proto$\left._j\right)$. Consequently, we can say that a packet $p_j$ is associated with the flow $\mathcal{F}_i$ if and only if: $k_i=k p_j$ and $t_i^{\text {start}} \leq t p_j \leq t_i^{\text {end}}$.

The procedure for constructing $\mathcal{H}$ is detailed in Algorithm 1. Although the basic definition of this structure is expressed as a single mapping $\mathcal{H}(\mathrm{k})$, the implementation promotes it into forward and backward indices to establish reliable processing of bidirectional network traffic. More specifically, each flow $\mathcal{F}_i$ is associated with dual directional keys: $k_{f w d_i}=$ $\left(\right.$srcIp$_i$, dstIp$_i$, srcPort$_i$, dstPort$_i$, proto$\left._i\right)$ for forward direction and $k_{\text {bwd}_i}=\left(\right.$dstIp$_i$, srcIp$_i$, srcPort$_i$, dstPort$_i$, proto$\left._i\right)$ with respect to backward one. Therefore, the temporal index $\mathcal{H}$ is reformulated to include two mapping functions: $\mathcal{H}\left\lceil k_{f w d}\right\rceil$ and $\mathcal{H}\left[k_{b w d}\right]$. Accordingly, Algorithm 1 initially generates for each flow a forward 5-tuple key and a backward 5-tuple key. Next, it inserts the relevant flow record $\left(t_i^{\text {start}}, t_i^{\text {end}}, i\right)$ into both indices: $\mathcal{H}\left\lceil k_{f w d}\right\rceil$ and $\mathcal{H}\left[k_{b w d}\right]$. Later, it sorts each flow list $L_k$ with respect to flow start time. Finally, it initializes a temporal indicator $\mathfrak{D}$ for each key, to enable straightforward access to filtered flow candidates associated with a packet.

Lemma 1. Let each flow ${{\mathcal{F}}_{i}}$ be assigned bidirectional indexing keys: a forward key $k_{i}^{fwd}$ and a backward key $k_{i}^{bwd}$ and stored into corresponding hash structure ${{\mathcal{H}}_{fwd}}$ and ${{\mathcal{H}}_{bwd}}$, respectively. Then, for any packet ${{p}_{j}}$ pertaining to flow ${{\mathcal{F}}_{i}}$, disregarding its direction, there is a direct retrieval path, such that ${{\mathcal{F}}_{i}}$ can be efficiently retrieved from either ${{\mathcal{H}}_{fwd}}$ or ${{\mathcal{H}}_{bwd}}$ utilizing the packet key.

Proof. As a flow ${{\mathcal{F}}_{i}}$ denotes bidirectional communication between a pair of endpoints, then a packet ${{p}_{j}}$ associated with that flow if and only if the key of ${{p}_{j}}$ is equal to either:

•the forward direction key $k_{i}^{fwd}~$of ${{\mathcal{F}}_{i}}$, thus this flow is identified through ${{\mathcal{H}}_{fwd}}$,

•or the backward direction key $k_{i}^{bwd}~$of ${{\mathcal{F}}_{i}}$, hence this flow is determined through ${{\mathcal{H}}_{bwd}}$.

As a consequence, the correct flow in both cases can be directly accessed without needing any key conversion or reverse lookup.

3.2 Packet-to-flow association

This stage is based on a packet-to-flow association algorithm (Algorithm 2), which is designed to process packets in a chronological order. Typically, it utilizes the hash index constructed in Algorithm 1 to identify the set of flows that remain active upon packet arrival. A key challenge is eliminating redundant traversal of expired or inactive flows. This is addressed by employing a monotonically advancing pointer, also produced by Algorithm 1, to selectively inspect flows whose temporal intervals may intersect the packet timestamp. The following formal definitions reflect that behavior.

Definition 5. Let $L_{{{k}_{p}}}^{d}$=$\left( t_{i}^{start},~t_{i}^{end},~i \right)$ denotes the ordered flows list associated with the packet key ${{p}_{k}}$ under $d$ direction, where $d\in \left\{ fwd,~bwd \right\}$. Consider ${{t}_{p}}\text{ }\!\!~\!\!\text{ }$is the packet timestamp and $\mathfrak{D}_{{{k}_{p}}}^{d}$is the corresponding directional indicator. Accordingly, the indicator increment operation is formulated as:

$\mathfrak{D}_{{{k}_{p}}}^{d}=\text{min}\{j~:j<\left| \text{ }\!\!~\!\!\text{ }L_{{{k}_{p}}}^{d} \right|\wedge t_{j}^{end}\ge {{t}_{p}}\}$. That is, the indicator is increased to the first flow whose end time is strictly greater than or equal to the packet timestamp, to effectively ignoring all expired flows.

Definition 6. The candidate active flow set associated with packet $p$ can be defined as follows: ${{A}_{p}}=\left\{ L_{{{k}_{p}}}^{d}\left[ j \right]:j\ge \mathfrak{D}_{{{k}_{p}}}^{d},\text{ }\!\!~\!\!\text{ }t_{j}^{start}\le {{t}_{p}}\le \text{ }\!\!~\!\!\text{ }t_{j}^{end} \right\}$. Therefore, ${{A}_{p}}$ comprises non-expired flows whose duration intervals incorporate the timestamp of packet $p$.

Specifically, given a packet $p~$in Algorithm 2, the associated flow list $L_{{{k}_{p}}}^{d}$is efficiently retrieved via constant time hash indexing. The indicator $\mathfrak{D}_{{{k}_{p}}}^{d}$ is then advanced to bypass expired flows whose end times exceed the packet timestamp. The remaining candidate active flows are sequentially examined until a temporal ordering condition is not satisfied, i.e., the flow start time precedes the packet timestamp.

Illustrative Example: This example elucidates how the hash structure $\mathcal{H}$ gathers flows sharing the same network key into sorted lists, thereby minimizing the packet-to-flow association problem from an exhaustive search across the entire set of flows to a more focused search within a restricted subset. Assume that three flows have been extracted from PCAP file, as illustrated in Table 1.

Table 1. Sample flows extracted from a packet capture file

Based upon this table, an associated key will be established for every flow, i.e, ${{k}_{fw{{d}_{1}}}}=\left( A,B,~2000,~60,~TCP \right),~~{{k}_{bw{{d}_{1}}}}=\left( B,~A,~60,~2000,~TCP \right)$ which are relevant to ${{\mathcal{F}}_{1}}$ and ${{\mathcal{F}}_{2}}$, and ${{k}_{fw{{d}_{2}}}}=\left( C,D,~1000,~500,~TCP \right)$ and ${{k}_{bw{{d}_{2}}}}=\left( D,~C,500,~1000,~TCP \right)~$which are associated with ${{\mathcal{F}}_{3}}$. Now, the flows can be organized within the hash structure to be as follows:

${{\mathcal{H}}_{fwd}}\left( {{k}_{fw{{d}_{1}}}} \right)=\left[ \left( 1,12,~{{\mathcal{F}}_{1}} \right),~\left( 6,18,~{{\mathcal{F}}_{2}} \right) \right]$

${{\mathcal{H}}_{bwd}}\left( {{k}_{bw{{d}_{1}}}} \right)=\left[ \left( 1,12,~{{\mathcal{F}}_{1}} \right),~\left( 6,18,~{{\mathcal{F}}_{2}} \right) \right]$

${{\mathcal{H}}_{fwd}}\left( {{k}_{fw{{d}_{2}}}} \right)=\left[ \left( 3,15,~{{\mathcal{F}}_{3}} \right) \right]$

${{\mathcal{H}}_{bwd}}\left( {{k}_{bw{{d}_{2}}}} \right)=\left[ \left( 3,15,~{{\mathcal{F}}_{3}} \right) \right]$

The forward and backward hash structures may contain identical flow record entries in order to serve distinct access paths and to enable constant-time lookup for packets regardless of their direction, without requiring key transformation at runtime.

In addition bidirectional lists are formed, $L_{{{k}_{1}}}^{fwd}=\left[ \left( 1,12,~{{\mathcal{F}}_{1}} \right),~\left( 6,18,~{{\mathcal{F}}_{2}} \right) \right]$, $L_{{{k}_{1}}}^{bwd}=\left[ \left( 1,12,~{{\mathcal{F}}_{1}} \right),~\left( 6,18,~{{\mathcal{F}}_{2}} \right) \right]$ and $L_{{{k}_{2}}}^{fwd}=\left[ \left( 3,15,~{{\mathcal{F}}_{3}} \right) \right],~L_{{{k}_{2}}}^{bwd}=\left[ \left( 3,15,~{{\mathcal{F}}_{3}} \right) \right]$. Both lists contain the same flow intervals but are indexed using different directions.

To apply Algorithm 2, assume that a packet $\text{p}$ has a timestamp ${{t}_{p}}=8,$ and key ${{k}_{p}}=\left( A,~B,~2000,~60,~TCP \right)$. According to this key and after checking the temporal conditions, it can be deduced that this packet matches both ${{\mathcal{F}}_{1}}$ and ${{\mathcal{F}}_{2}}$, since the packet timestamp is within the intervals of both flows. However, when the same packet has a timestamp equal to 14, then the indicator bypasses ${{\mathcal{F}}_{1}}$ as its end time precedes 14, remaining only ${{\mathcal{F}}_{2}}$ as an active flow. This confirms the effectiveness of the indicator mechanism in omitting redundant traversals over inactive flows while preserving sound temporal matching.

Since multiple flows might have the same direction and may be simultaneously ready for the same packet, a deterministic selection algorithm (Algorithm 3) is introduced. Initially, this algorithm discards candidate flows for which the directional packet capacity $cap_{f}^{d}~$is depleted.$cap_{f}^{d}$ refers to the number of packets that remain assignable to a flow in a specific direction, based on its reference forward or backward packet count. When only one eligible candidate remains, it is chosen directly. In cases where multiple candidates remain, the algorithm prefers an already-initiated flow whose timing falls within the bounds of the reference duration, and selects the most recently active flow. In the absence of such a flow, the candidate whose reference time window is closest to the packet timestamp is chosen instead. When several candidate flows have an equal temporal distance from the packet,$di{{s}_{f}}$, priority is granted to the flow with the earlier reference time window. If the candidates still have the same referential windows, the flow with the smallest reference index is used as an ultimate deterministic tiebreaker. As a result, each packet maps to at most one flow, guaranteeing duplication-free and deterministic packet-to-flow association.

3.3 Feature augmentation using payload-based statistics

During this stage, each flow undergoes a feature augmentation process by incorporating of discriminative statistical features extracted from its relevant packet payload contents. In contrast to conventional flow-level features, which mainly depend on packet header information, the proposed framework consolidates statistical measures calculated directly from raw payload data to attain fine-grained traffic behavior. In essence, this stage aims to capture intrinsic patterns within the data distribution, thereby enhancing the distinction between benign traffic and anomalous traffic, particularly in scenarios involving encrypted or obfuscated communications.

Typically, this stage consists of four sequential steps, as described below. In the first step, the payload for every packet belonging to a particular flow is represented as a sequence of bytes. In the second step, three statistical metrics are calculated for this sequence.

•Shannon Entropy: In the packet context, entropy denotes the degree of uncertainty in the payload byte distribution. It can be considered an indicator of payload randomness, since encrypted, compressed, or obscured content may exhibit higher entropy values than structured and regular traffic. Given a payload with byte probability distribution $Prob\left( b \right)$, the entropy can then be computed as:

$E\left( {{p}_{j}} \right)=-\underset{b}{\mathop \sum }\,Prob\left( b \right){{\log }_{2}}Prob\left( b \right)$     (1)

•Chi-Square Deviation: It can be defined in this study as a measurement that determines whether the actual byte-frequency distribution considerably deviates from the estimated equal-probability distribution according to the following formula:

$CSD\left( {{p}_{j}} \right)=\underset{b}{\mathop \sum }\,\frac{{{\left( {{A}_{b}}-{{E}_{b}} \right)}^{2}}}{{{E}_{b}}}$    (2)

where, ${{A}_{b}}$ represents the actual distribution of byte value $b$ contained in the payload of packet ${{p}_{j}}$, and ${{E}_{b}}$ indicates the estimated (predicted) distribution of byte value $b$ under an even distribution. Therefore, the chi-square deviation can quantify the extent of deviation from uniformity, with lower values indicating a more uniform byte distribution and higher values indicating greater deviations from uniformity.

•Bit Balance: It is used to describe the distributional characteristics of the binary value distribution within a payload as it is defined as:

$BB\left( {{p}_{j}} \right)=\frac{{{N}_{1}}}{{{N}_{bits}}}$         (3)

where, ${{N}_{1}}$ denotes the total count of occurrences of bits equal to 1, and ${{N}_{bits}}$ is the aggregate number of ones and zeros in payload ${{p}_{j}}$. Payloads with bit balance values approaching 0.5 primarily manifest arbitrary-like behavior, and biased distribution, though, may refer to structured or deterministic content.

In the third step, the calculated payload features are aggregated for each flow to construct a concise representation, due to the multiple packets contained within every single flow. For each feature $\text{ }\!\!\Upsilon\!\!\text{ }\in \left\{ E,~CSD,~BB \right\}$, and for each packet ${{p}_{j}}\in {{F}_{i}}$, four statistical descriptors are calculated, including mean $\text{ }\!\!\mu\!\!\text{ }$, standard deviation $\text{ }\!\!\sigma\!\!\text{ }$, maximum $max$, and minimum $min$ values.

In the final step, for each flow ${{F}_{i}}$, a final feature vector is formed as:

$\gamma \left( {{F}_{i}} \right)=\{{{\mu }_{E}},~{{\mu }_{CSD}},{{\mu }_{BB}},~{{\sigma }_{E}},~{{\sigma }_{CSD}},~{{\sigma }_{BB}},~ma{{x}_{E}},~ma{{x}_{CSD}},~$

$ma{{x}_{BB}},~mi{{n}_{E}},~mi{{n}_{CSD}},~mi{{n}_{BB}}\}$

This feature vector is finally concatenated with the original preexisting flow-based vector to produce the comprehensive input to the classification model.

3.4 Machine learning-based classification stage

After producing the augmented feature space, supervised learning models are employed to assess the effectiveness of the proposed model. In essence, five classification models representing different learning paradigms are chosen: logistic regression [26], extra trees [27], adaptive boosting [28], extreme gradient boosting [29], and a multilayer perceptron [30]. These classifiers are chosen to cover all four types of learning paradigms: linear, ensemble, boosting and neural-network.

3.5 Shapley Additive Explanations-based interpretability stage

In order to improve the interpretability of the selected classifier, SHAP is utilized. In principle, SHAP characterizes the contribution of both traditional flow-level features and the introduced payload-derived features to the final detection decision, and evaluates the effectiveness of these features.

This section introduces and discusses the experimental results generated from applying the proposed framework. Essentially, the results are discussed from the following three perspectives:

5.1 Packet-to-flow association performance

The packet-to-flow association algorithms are evaluated using the CICIDS2017 dataset, which comprises large-scale and heterogeneous daily traffic captures wherein both packet captures and flow records co-occur within the same network trace. This dataset requires explicit packet-to-flow association processing prior to integrating the payload-oriented features with the flow-level ones. Conversely, the adopted CICIoMT2024 subsets are structured into separate packet capture and CSV files for each attack category, without explicit flow identifiers, enabling payload-extracted features to be combined via protocol-aware aggregation rather than direct packet-to-flow associating.

In order to evaluate the correctness of the packet-to-flow-association algorithm, Table 2 summarizes a set of complementary validation metrics encompassing both packet and flow levels. For packet-level analysis, the time-eligible packets metric is used to identify which packets have timestamps that lie within the temporal range mentioned by the reference CSV records. Eligible association coverage quantifies the percentage of eligible packets successfully associated with flows and is calculated using the following equation:

$Eligible~Assocattion~Coverage=\frac{{{N}_{assigned}}}{{{N}_{eligible~}}}$                      (4)

where, ${{N}_{assigned}}$ is the number of assigned eligible packets, and ${{N}_{eligible~}}$is the total number of time eligible packets. In addition, flow-level packet-count consistency was assessed by defining the absolute packet-count difference for each flow ${{F}_{i}}$:

                  ${{D}_{i}}=\left| {{N}_{assigne{{d}_{i}}}}-{{N}_{CS{{V}_{i}}}} \right|$           (5)

where, ${{N}_{assigne{{d}_{i}}}}$ denotes the number of packets assigned by the proposed association algorithm to flow ${{F}_{i}}$, and ${{N}_{CS{{V}_{i}}}}\text{ }\!\!~\!\!\text{ }$represents the original packet count related to flow ${{F}_{i}}$ and reported in the CSV file. Depending on ${{D}_{i}}$, exact match rates and agreement rates within tolerance margins of 2 and 5 packets, i.e., $\left| {{N}_{assigned}}-{{N}_{CSV}} \right|\le 2,~~\text{and}~\left| {{N}_{assigned}}-{{N}_{CSV}} \right|\le 5$, respectively, were then computed, together with the mean and median of ${{D}_{i}}\text{.}$ Precise packet-count agreement was defined as the proportion of flows for which the number of packets assigned by the proposed algorithm exactly matched the corresponding packet count recorded in the reference CSV file. Depending on ${{D}_{i}}$, agreement rates within tolerance margins of 2 and 5 packets, i.e.,$\left| {{N}_{assigned}}-{{N}_{CSV}} \right|\le 2,~and~\left| {{N}_{assigned}}-{{N}_{CSV}} \right|\le 5$, respectively, were then computed, together with the mean and median of ${{D}_{i}}\text{.}$

The temporal analysis results revealed that, out of the 11,497,247 Transmission Control Protocol (TCP)/User Datagram Protocol (UDP) packets inspected, 3,082,997 packets (26.82%) fell outside the temporal range covered by the reference CSV records. As a result, these packets were excluded from the association evaluation instead of being considered as failed associations. The remaining 8,414,250 packets constituted the temporally eligible group, of which 8,370,404 were successfully associated with their respective flows. Consequently, the eligible association coverage attained 99.48%, with only 0.52% of the eligible packets failing to be associated. Precise packet-count agreement was recorded for 50.33% of the evaluated flows, indicating the absence of any difference between the reconstructed and reference packet counts. Considering minor deviations, 68.06% of the flows deviated by at most 2 packets, while 86.40% deviated by at most 5 packets. These tolerance-based metrics permit a graded assessment of discrepancy magnitude, instead of treating all nonzero differences as equivalent. The median packet-count difference or the central value of the observed differences was found to be zero packets. Conversely, the mean absolute packet-count difference amounted to 6.94 packets, implying that a smaller subset of flows with notably larger discrepancies skewed the overall average upward. It is important to note that the 99.48% value represents the extent of association coverage instead of the accuracy of individual packet-level associations; it should not be considered as association correctness. In the same vein, packet-count agreement captures numerical consistency, but does not guarantee, by itself, the correctness of individual packet-to-flow mappings. These additional validation findings support the effectiveness of the deterministic disambiguation selection strategy presented in Algorithm 3. Typically, out of 3,373,803 packets exhibiting multiple initial flow candidates, 1,099,747 were reduced to a single candidate via the directional-capacity constraint, whereas the remaining 2,253,348 ambiguous cases were disambiguated by conducting the deterministic temporal rule. Notably, neither duplicate packet assignments nor directional-capacity overruns were detected, confirming that each packet was associated with at most one flow.

Table 2. Experimental validation of the proposed packet-to-flow association algorithm

As a complement to the automated validation, a manual audit was conducted on a sample of 20 packet-to-flow associations [35] randomly selected from the TCP/UDP packets within the Tuesday-WorkingHours packet capture. A stratified sampling strategy was applied to capture both association outcomes relevant to the proposed algorithm, consisting of 10 DIRECT-UNIQUE cases featuring a single eligible flow candidate and 10 DISAMBIGUATED cases involving multiple overlapping candidates. This strategy ensures that the audit addressed both straightforward associations and the more difficult cases that called for deterministic resolution. Each sampled packet was manually traced back to its original packet capture entry and compared with the corresponding reference-flow record, checking the connection identifiers, direction, and the timing for consistency. An association was treated as correct only when the reference information clearly and uniquely underpinned the selected flow; it was deemed as incorrect if the reference evidence unambiguously indicated a different flow; otherwise, it was labeled as indeterminate to avoid overstating accuracy. The findings were dissimilar between the two audit strata (Table 3). Each of the 10 DIRECT-UNIQUE cases was judged separately as correct, with no cases deemed incorrect or indeterminate. Conversely, all 10 DISAMBIGUATED cases stayed indeterminate due to the inadequacy of the accessible reference information to independently identify a single candidate flow. Therefore, the ambiguity-resolution decisions of Algorithm 3 were not independently verified by the manual audit that was considered a conservative audit.

The outcomes of the manual audit were not the same as the 99.48% association coverage reported earlier. The coverage value represented the percentage of temporally eligible packets that were successfully associated with flows, whereas the manual audit assessed whether the selected associations could be independently confirmed. Thus, the 99.48% coverage must not be regarded as an indication of the association's correctness.

Table 3. Manual audit findings organized by association stratum

5.2 Session-based partitioning performance

To test the impact of the inactivity threshold on session formation and data partitioning, four threshold values (1, 5, 10, and 15 minutes) were analyzed. Table 4 presents a summary of the session features and the stability of the partitioning.

Table 4. Sensitivity of data partitioning-based session awareness to the inactivity threshold

In general, the sensitivity study indicates that the session-based partitioning exhibited stability across all assessed inactivity threshold values. No overlap in sessions was seen among the training, validation, and test subsets for any evaluated value of $\text{ }\!\!\alpha\!\!\text{ }$, while the resultant partition sizes closely adhered to the anticipated 60/20/20 distribution. As the threshold value rose, the number of sessions decreased while the average flow count per session grew, indicating the anticipated consolidation of temporally nearby flow records.

Following this sensitivity analysis, a 5-minute inactivity threshold was used for the final evaluation experiments as a moderate parameter setting between the extremes of excessive session fragmentation and the stronger consolidation noted with longer thresholds. It further preserved the desired 60/20/20 partition ratios and was therefore applied for the classification findings reported in Tables 5–10.

An independent record-level audit was conducted to supplement the session-level examination. Prior to the construction of sessions and the partitioning of data, each flow record was allocated a unique original-row identification, which was preserved exclusively for auditing purposes and omitted from the predictive feature set. Subsequent to partitioning, the identifier sets were evaluated in pairs across the training, validation, and test subsets. No overlapping original-record identifiers were detected among training and validation, training and test, or validation and test, as the intersection sizes yielded zero for these three comparisons. This record-level evaluation was documented independently from session overlap, as the absence of session overlap was an anticipated characteristic of group-based partitioning and did not inherently confirm independence at the capture, day, or source level.

5.3 Classification performance

Regarding the CICIDS2017 dataset, the assessed classifiers exhibited commendable detection efficacy in both binary and multi-class scenarios under the session-based partitioning algorithm. Tables 5 and 6, along with Figures 2 and 3, compare the suggested payload-oriented attributes with the conventional flow-level depiction. To address class imbalance, macro-averaged precision, recall, and F1-score were utilized, ensuring each class contributed equally irrespective of its prevalence. In the context of binary classification, extreme gradient boosting attained the superior overall performance, recording an accuracy of 0.99991, macro-precision of 0.99891, macro-recall of 0.99917, macro-F1 score of 0.99904, and area under the receiver operating characteristic curve (ROC-AUC) of 1.00000. Adaptive boosting also yielded commendable outcomes, especially regarding macro-F1 (0.99726) and ROC-AUC (1.00000). In the multi-class classification case, extreme gradient boosting had the most robust performance, attaining an accuracy of 0.99994, macro-precision of 0.99998, macro-F1 score of 0.99869, and ROC-AUC of 1.00000. Adaptive boosting produced results that were nearly equivalent and achieved a marginally superior macro-recall (0.99792) compared to extreme gradient boosting (0.99741). The other classifiers exhibited diminished macro-F1 scores, especially in the multi-class context, even if they often had elevated accuracy and ROC-AUC metrics. In summary, these findings suggest that extreme gradient boosting and adaptive boosting exhibit the best reliable performance across the assessed metrics under the utilized session-based partitioning framework.

Table 5. Classification performance of the proposed payload-oriented feature representation on the CICIDS2017 dataset

Table 6. Classification performance of the flow-oriented feature representation on the CICIDS2017 dataset

2.png

Figure 2. Macro-F1 score comparison on CICIDS2017 based on binary classification

3.png

Figure 3. Macro-F1 score comparison on CICIDS2017 based on multi-class classification

The introduced payload-driven feature representation attained impressive results on the CICIoMT2024 dataset in terms of binary and multi-class classification tasks, as presented in Table 7. For binary classification, extreme gradient boosting demonstrated superior performance, with an accuracy of 0.98997 and a macro-F1 of 0.97061, whereas adaptive boosting had a macro-F1 of 0.89549. The remaining classifiers recorded macro-F1 scores of 0.88856, 0.86227, and 0.85364 for the multilayer perceptron, extra trees, and logistic regression, respectively. Within the multi-class case, extreme gradient boosting again attained superior performance, with an accuracy of 0.99036 and a macro-F1 score of 0.93017. Reduced macro-F1 scores were noted for the alternative classifiers, especially extra trees and logistic regression, signifying variances in their ability to differentiate among various traffic categories when equal weight was given to each category. In general, extreme gradient boosting had the most reliable performance in both classification tests.

In contrast, Table 8 displays the findings achieved by the original flow-based feature representation. Extreme gradient boosting recorded macro-F1 scores of 0.95412 and 0.88623 for binary and multi-class classification, respectively. The addition of the recommended payload-focused attributes elevated the macro-F1 score across all five classifiers in both cases. The macro-F1 metric in binary classification improved from 0.95412 to 0.97061 for extreme gradient boosting, from 0.84279 to 0.89549 for adaptive boosting, from 0.80857 to 0.88856 for multilayer perceptron, from 0.85875 to 0.86227 for extra trees, and from 0.77656 to 0.85364 for logistic regression. The enhancements were predominantly more significant in the multi-class task: macro-F1 rose from 0.88623 to 0.93017 for extreme gradient boosting, from 0.64202 to 0.76567 for adaptive boosting, from 0.63048 to 0.70317 for multilayer perceptron, from 0.59455 to 0.62109 for extra trees, and from 0.53299 to 0.60082 for logistic regression. These findings suggest that the payload-centric descriptors offer additional distinguishing information beyond the initial flow characteristics, with the impact of the payload-focused descriptors remaining apparent in the more intricate multi-class classification scenario.

To provide additional insight into the effectiveness of the features extracted in this study, Figures 4 and 5 present a performance comparison of the flow-based and payload-based feature representations under binary and multi-class classification settings, respectively. The comparison is based on the macro-F1 score metric because effective intrusion detection requires high detection capability while minimizing false alarms, which are concurrently captured by this metric via the joint use of precision and recall.

Table 7. Classification performance of the proposed payload-oriented feature representation on the CICIoMT2024 dataset

Table 8. Classification performance of the flow-oriented feature representation on the CICIoMT2024 dataset

4.png

Figure 4. Macro-F1 score comparison on CICIoMT2024 based on binary classification

5.png

Figure 5. Macro-F1 score comparison on CICIoMT2024 based on multi-class classification

Table 9. Feature-level ablation results for binary classification with test macro-F1 scores (mean ± standard deviation)

Table 10. Feature-level ablation results for multi-class classification with test macro-F1 scores (mean ± standard deviation)

The feature-level ablation assessment conducted over five random seeds, including 42, 52, 62, 72, and 82, indicates that the impact of the payload-derived descriptors differs among classifiers and classification tasks (Tables 9 and 10). Essentially, in binary classification, extreme gradient boosting exhibited consistently superior performance across all configurations, achieving macro-F1 values near 0.9995 with minimal fluctuations across seeds. A more substantial effect was recorded for extra trees, as its macro-F1 escalated from 0.92185 ± 0.00521 with the original flow characteristics to 0.98637 ± 0.00679 after integrating all three payload descriptors. Within the multi-class scenario, extreme gradient boosting maintained impressive stability, recording 0.99931 ± 0.00045 with the basic features and 0.99944 ± 0.00052 with the whole feature set. The influence of the payload descriptors was notably obvious for adaptive boosting, where the macro-F1 advanced from 0.97011 ± 0.00403 to 0.99910 ± 0.00070 when all three descriptors were utilized. Alternatively, extra trees showed enhanced variability in relation to feature configurations and random seeds, signifying a stronger sensitivity to feature composition and partitioning of data. The repeated-seed outcomes reveal that Shannon entropy, chi-square deviation, and bit balance deliver supplementary data, whose influence is reliant on the classifier instead of achieving a standardized improvement across all models.

5.4 Shapley Additive Explanations-based explainability analysis

The SHAP analysis provides insights into the contributions of both traditional flow-level features and payload-derived features. Essentially, the SHAP analysis depicted in Figure 6 for CICIDS2017 demonstrates that the extreme gradient boosting model is mostly driven by standard flow attributes, notably Bwd Header Length, Packet Length Mean, and Init_Win_bytes_forward. Of the recommended payload-derived descriptors, EntropyFwd_Count ranks fourth in global feature importance, suggesting that payload entropy contributes additional information alongside the prevailing flow-based attributes. In the context of CICIoMT2024, Figure 7 illustrates that traditional traffic characteristics, specifically IAT, rst_count, and Rate, possess the greatest global SHAP significance. Nevertheless, other suggested payload-derived descriptors are also listed among the highest-ranked attributes, such as EntropyBwd_Count, ChiSquareFwd_Min, EntropyFwd_Count, ChiSquareFwd_Max, and EntropyBwd_Min. This suggests that entropy- and chi-square-derived payload descriptors offer supplementary distinguishing insights compared to traditional traffic characteristics.

6.png

Figure 6. Shapley Additive Explanations (SHAP) analysis of the proposed payload-oriented features on the CICIDS2017 dataset

7.png

Figure 7. Shapley Additive Explanations (SHAP) analysis of the proposed payload-oriented features on the CICIoMT2024 dataset