Instance classification encounters several problems, such as large number of features. In order to improve classification accuracy and reduce computation times, we need to reduce the dimensionality of the data. For that end, we use Deep Auto-Encoder (DAE) [13] . Contrary to the conventional Ordinary Auto-Encoder (OAE) with a single hidden layer, the Deep Auto-Encoder (DAE) depicted in Figure 1 utilizes multiple hidden layers.

The hidden layers link the input layer with the output layer. They are stacked one after the other to form the encoder with the input layer, and the decoder with the output layer. In order to extract the strongest and most important features for proper classification, the DAE goes through two main phases: encoder and decoder. The encoding phase takes place between the input layer and the hidden layers up to the bottleneck layer. In this phase, the input data is transformed into reduced-dimensional representation, while retaining relevant features.

To train the autoencoder, we began by determining the number of hidden layers and neurons per layer, monitoring the risk of overfitting. Next, we selected the activation function, opting for one that offers both speed and stability. Regarding the cost function, in line with the literature [13] recommending mean squared error (MSE) for reconstructions, we adopted this approach. For the learning rate, we started with a low value, which we gradually adjusted to avoid overly slow convergence. We then determined the appropriate number of epochs and batch size to stabilize the model. The final step was to assess the quality of the reconstruction on a validation set, which led us to adjust parameters such as the number of layers, the learning rate, and the batch size.

The encoder evaluates (formula 1) vector h (latent or bottleneck vector) with size m from the input vector X which represents the original data.

$h=f\left(W^{\left(1\right)}X+b^{\left(1\right)}\right)$(1)

where: $W^{\left(1\right)}$ is a matrix $m\times n$, $b^{\left(1\right)}$ is a bias vector of size n and $f\left(\right)$ is an encoder activation function.

After that, came the decoder phase. It takes place between the hidden layers (starting with the bottleneck layer) and the output layer which has the same number of nodes as the input layer. The hidden layers in this phase are symmetrical to the hidden layers in the previous phase, with one in common (the bottleneck layer). The DAE try to have an output vector $\hat{X}$ that is closet possible representation of the initial attributes (those of the input layer: X), by minimizing the loss function. In this work loss function is mean squared error loss function (formula (2)).

$\text{Loss}=\frac{1}{n}\left({\displaystyle {\sum }_{i=1}^n{\left(X_i-{\hat{X}}_i\right)}^2}\right)$(2)

The decoder tries to build output vector from the latent vector h, the result of which is the vector $\hat{X}$.

$\hat{X}=g\left(W^{\left(2\right)}h+b^{\left(2\right)}\right)$(3)

where $W^{\left(2\right)}$ is a matrix $n\times m$, $b^{\left(2\right)}$ is a bias vector of size n, $\hat{X}$ is a vector of size m, a reconstruction of the input vector X as close as possible to it.

After remarkably reduced the number of attributes (features) in the original database, in this section, we propose to predict the classes of the new instances, for which we’ve chosen to apply Bidirectional Long Short-Term Memory (BiLSTM) [14] . BiLSTM architecture is an extension of recurrent neural networks (RNN), designed to capture temporal dependencies in data sequences in both past and future directions. It is composed with two Long Short-Term Memory (LSTM) [15] .

A sequence of LSTM [16] is made up of cells connected across time steps t. Each cell’s output is regulated by a group of activation functions $\left\{f\left(t\right),i\left(t\right),o\left(t\right)\right\}$, referred to as gates [17] , as depicted in Figure 2 . These gates yield values in ${ℝ}^T$, where T represents the dimension (the number of cells in the sequence) of the LSTM. The function $f\left(t\right)$, often referred to as the forget gate, determines the degree to which the information from the preceding cell $c_{t-1}$ is discarded [18] . The input gate $i\left(t\right)$ specifies the portion of information to be stored in cell $c_t$. Meanwhile, the output gate $o\left(t\right)$ governs the fraction of the internal state transmitted to the subsequent cell.

The cell input $x\left(t\right)$ at time t concatenates with output of a cell $c\left(t-1\right)$ at previous time step $\left(t-1\right)$ which is $h\left(t-1\right)$. The resultant vector traverses through the input node and passes through the gates responsible for input, forgetting, and output to give $h\left(t\right)$. Evaluation of $h\left(t\right)$ (Equation (4.6)) for each cell can be find in many papers, for example in [19] and [20] .

$f\left(t\right)=\sigma \left(w_{fh}h\left(t-1\right)+w_{fx}x\left(t\right)+b_f\right)$ (4.1)

$i\left(t\right)=\sigma \left(w_{ih}h\left(t-1\right)+w_{ix}x\left(t\right)+b_i\right)$(4.2)

$\tilde{c}\left(t\right)=\tanh \left(w_{\tilde{c}h}h\left(t-1\right)+w_{\tilde{c}x}x\left(t\right)+b_{\tilde{c}}\right)$(4.3)

$c\left(t\right)=f\left(t\right)\cdot c\left(t-1\right)+i\left(t\right)\cdot \tilde{c}\left(t\right)$(4.4)

$o\left(t\right)=\sigma \left(w_{oh}h\left(t-1\right)+w_{ox}x\left(t\right)+b_o\right)$(4.5)

$h\left(t\right)=o\left(t\right)\cdot \tanh \left(c\left(t\right)\right)$(4.6)

It is important to notice that, there is many variants of LSTM cell depend on how function $\sigma$ and tanh are arranged and their number in the cell [18] . An LSTM layer operates unidirectionally, processing input sequentially from left to right, thus encoding dependencies solely on preceding elements in the sequence. To address this limitation, we employ an additional LSTM layer that operates in the opposite direction, enabling the detection of dependencies on subsequent elements in the text by processing input from right to left. This arrangement gives rise to a neural network known as a Bidirectional LSTM (BiLSTM) [21] . It enables the network to assign equal significance to both the initial and final segments of the sequence, leading to enhanced performance.

Additionally, we illustrate an instance of a bidirectional LSTM layer in Figure 3 . The result of a BiLSTM layer is the concatenation of the outputs in both directions, as show in Equation (5)

$H=\left[\stackrel{\rightharpoonup }{h_T},\stackrel{\leftarrow }{h_T}\right]$(5)

where $\stackrel{\rightharpoonup }{h_T}$ and $\stackrel{\leftarrow }{h_T}$ are define by Equation (4.6).

The intrusion detection method we propose is divided into two steps. The first step involves dimensionality reduction using Deep Auto-Encoder (DAE), followed by a phase of behavior class prediction with BiLSTM recurrent neural networks. The dimensionality reduction process begins with data cleaning, which involves identifying, correcting, or removing erroneous data. Errors are indicated by missing values (NaN), infinite values (Inf), and empty cells within the dataset. To address these issues, we have developed an algorithm that replaces NaN values with 0, substitutes Inf values with the maximum value of the dataset, and fills empty cells with the mean value of the relevant attributes.

Following data cleaning, we moved on to normalization. This preprocessing step adjusts the data values to ensure they fall within a consistent range, which helps machine learning algorithms function more efficiently and reliably. In this study, we applied Z-Score normalization) as our chosen technique. This method standardizes the data to have a mean of 0 and a standard deviation of 1, as specified in Equation (6).

$X^{\prime }=\left(X-\mu \right)/\sigma$ (6)

where X represents the original data value, X’ is the normalized value, with $\mu$ and $\sigma$ representing the mean and standard deviation, respectively. Z-Score normalization is the most appropriate method for autoencoders [22] as it provides a uniform and centered scale for the data, which enhances the efficiency of the autoencoder’s reconstruction and the model’s stability. The final step in this phase was to design and train the deep autoencoder architecture to extract reduced and representative features from the original data. Algorithm 1 below outlines the sequence of steps involved in the dimensionality reduction process.

Algorithm 1. Dimensionality reduction.

In line 12 and 13, $h\left(0\right)$ is the input data. $f(.)$ is activation function. $W\left(l\right)$ and $b\left(l\right)$ are the weights and biases of layer $l$.

The compressed data from Algorithm 1 is subsequently utilized in the classification phase, which is based on a Bidirectional LSTM, to determine if the traffic is normal or indicative of an attack. The steps of this process are outlined in Algorithm 2 .

Algorithm 2. Intrusion detection.

The experiments were carried out on a PC equipped with a 16 GB RAM Core i7 processor running Windows 11 64-bit. For software, we utilized Python 3.7 within the Anaconda integrated development environment (IDE), which serves as the editor and includes various libraries such as Scikit-learn, TensorFlow, Keras, SciPy, NumPy, Pandas, and Matplotlib.

Concerning the dataset, we use a most recent Canadian Institute for Cybersecurity (CIC) dataset called CIC2017 available at [23] . The CICIDS2017 dataset offers a wide range of attack scenarios, including Denial of Service (DoS), Distributed Denial of Service (DDoS), brute force, web attacks, infiltration, botnet, and others. This variety makes it ideal for testing and comparing different intrusion detection techniques. The dataset simulates realistic network traffic by combining both normal and malicious activities, closely resembling real-world conditions as opposed to fully synthetic datasets. It provides over 80 features extracted from network traffic, encompassing basic information like IP addresses and port numbers, as well as more advanced statistical metrics such as packet length and flow duration. Further details can be found at [24] .

As mentioned in our proposal, we have chosen deep auto-encoder as module for dimensionality reduction. After carrying out several executions, to find the right optimal value for the number hidden layers, we return 3 hidden layers between the input layer which is a vector of size 80 and the bottleneck layer whose size is 25. The total number of trainable parameters is 22,141. The learning curve for our deep autoencoder was set to 20 epochs with a batch size of 320 (after several evaluation tests). Table 1 summarizes the features of DAE.

The proposed DAE enabled us to reduce the number of attributes from 80 to 25 (i.e. a compression factor of 1:3.2). Figure 4 shows the graph of the loss function for the different number of epochs, and this in both train and test dataset. As the amount of loss is under 5 × 10⁻⁴ for epoch near 40. An output vector very close to the input vector is obtained from the 25 selected features. Then, BiLSTM help us to classify data instances with satisfactory accuracy.

We also conducted another experiment to evaluate the performance of our proposal. For this, we used two metrics: accuracy and recall. Our proposal (DAE-BiLSTM) performs better than random forests, AdaBoost, or BiLSTM (see Figure 5 ). DAE-BiLSTM achieved an accuracy of 97% and a recall of 95%, respectively.

Finally, we evaluated the ROC curves and AUC values of Random Forest, Adaboost, BiLSTM, and DAE-BiLSTM in an intrusion detection task. The results, shown in Figure 5 , indicate that DAE-BiLSTM achieved the highest AUC, close to 0.93, outperforming all other models.

Upon examining Figure 4 , we observe that the training and test loss curves gradually decrease before stabilizing around the 40^th epoch, reaching a loss rate significantly below 5 × 10⁻⁴. At this point, the two curves converge, indicating that the model is learning effectively from the training data while generalizing properly to the test data. The progressive decrease in loss followed by stabilization at a low value reflects the model’s good performance. This convergence toward a low loss value, with no sign of divergence between the two curves, suggests that there is neither underfitting nor overfitting. The model successfully captures the characteristics of the data without overfitting the training set, which is an indicator of its generalization capability.

With regard to the histogram in Figure 5 , which represents the models’ performance on Accuracy and Recall, we note that the DAE-BiLSTM model is the most performant for this intrusion detection problem, with higher performance on both metrics. The BiLSTM follows closely, while Adaboost and Random Forest are less effective. This indicates that, for this complex problem, recurrent neural networks are a top choice. More specifically, the fact that DAE-BiLSTM has the highest Accuracy (0.97) and Recall (0.95) among the four models means that it is not only capable of making much more accurate predictions than the others, but it is also excellent at correctly identifying positive cases. Therefore, it is a model that generalizes well.

The performance of the DAE-BiLSTM compared to other models can be explained by the fact that deep autoencoders improve data processing by reducing dimensionality, learning compact representations that highlight key features and filter out noise. This enhances the quality of input features for the BiLSTM and boosts detection accuracy. By focusing on essential characteristics, the DAE-BiLSTM reduces the risk of overfitting and improves generalization, enabling better identification of network attacks in new data. Lower input dimensionality also reduces computational costs, allowing faster training and more efficient resource use, while smoothing the optimization process and decreasing the likelihood of the model getting stuck in local minima, which potentially boosts accuracy. The refined features from the first step help the BiLSTM in the second step better distinguish between normal and malicious activities, reducing false positives and improving the overall reliability of the intrusion detection system.

Figure 6 also shows the DAE-BiLSTM’s superiority, as its Receiver Operating Characteristic (ROC) curve approaches the top-left corner, demonstrating outstanding intrusion detection accuracy with minimal errors. This model excels by combining the strengths of the Deep Autoencoder (DAE) for dimensionality reduction and the BiLSTM for capturing temporal patterns, resulting in exceptional performance. Even when used on its own, BiLSTM performs well, indicating that recurrent neural networks are effective for intrusion detection due to their ability to recognize temporal relationships in data. On the other hand, traditional models like Adaboost and Random Forest fall short, likely because they struggle to handle complex, high-dimensional data. Area Under the Curve (AUC) serves as a crucial metric for evaluating model performance, with higher values indicating better class separation. The DAE-BiLSTM leads the pack with an AUC of 0.93, followed by BiLSTM (0.86), Adaboost (0.76), and Random Forest (0.68). The success of DAE-BiLSTM lies in its dual approach: dimensionality reduction via the DAE, which filters noise and emphasizes important features, and temporal pattern recognition by the BiLSTM. This combination enhances the model’s generalization and reliability, making DAE-BiLSTM the optimal choice for intrusion detection tasks.