The IoT-23 dataset [8] is a new dataset of network traffic from Internet of Things (IoT) devices. It contains 20 captures of malware running on IoT devices and three captures of benign IoT device traffic. It was first published in January 2020, with captures spanning from 2018 to 2019. This IoT network traffic was captured in the Stratosphere Lab, AIC Group, FEL, CTU University, Czech Republic. Its goal is to provide a large dataset of real-world, labeled IoT malware infections and benign IoT traffic for researchers to develop machine learning algorithms. This dataset and its research are funded by Avast Software, Prague.

Data preprocessing is a crucial step in our architecture, combining an LSTM autoencoder (AE-LSTM) and a multi-agent reinforcement learning system (Multi-Agent PPO), applied to intrusion detection on the IoT-23 dataset.

The IoT-23 dataset consists of conn.log.labeled files, each corresponding to a specific network scenario (e.g., Mi-rai, infiltration, normal activity, etc.). To ensure optimal quality of the inputs processed by the autoencoder and RL agents, several preparation steps are necessary.

First, rigorous cleaning is performed. This consists of deleting rows without a detailed label (detailed-label), eliminating missing values (NaN), infinite values (Inf), and duplicates. A type conversion is then applied to transform all categorical columns into usable numeric formats. In addition, a correlation matrix analysis is performed, and all highly correlated columns (correlation > 0.95) are removed to avoid redundancy and reduce dimensionality.

Next, manual label mapping is performed to associate each type of network activity with a unique numeric identifier. This allows the autoencoder to distinguish between normal and abnormal behaviors and facilitates the interaction of RL agents with their environment based on rewards per type of detected behavior.

Normalization is then applied to all numeric variables using the StandardScaler method to center and reduce the data, ensuring stable convergence of the AE-LSTM model.

Finally, to ensure balanced exploration of the environment by the PPO agents, minority classes are resampled. Each class is balanced to contain a fixed number of examples (e.g., 1500) to ensure each PPO agent has a fair representation of the various network states to be analyzed. The final dataset, thus cleaned, converted, normalized, and balanced, is then transformed into time sequences to be injected into the AE-LSTM module, which is used to encode the dynamics of network flows. These representations are then used as input states for the PPO agents responsible for making optimal decisions for anomaly or attack detection.

1) Preprocessing Process Algorithm ( Figure 1 ):

Figure 2 illustrates the different steps of the proposed process for detecting intrusions in IoT networks. The system is based on a combination of an LSTM autoencoder for feature extraction and a set of PPO (Proximal Policy Optimization) agents operating in a multi-agent reinforcement learning environment. This innovative architecture is designed to learn optimal behaviors in response to different types of IoT traffic, based on the compact representations provided by the AE-LSTM model.

The processing of the IoT-23 dataset uses captures based on complete traffic scenarios (Pcap, Flows) rather than independent samples. To prevent data leakage during anomaly detection or classification, the following steps should be taken:

The splits were performed at the level of entire scenarios to avoid any data leakage when processing the IoT-23 dataset. The flows and time windows extracted from a given scenario are assigned exclusively to the training/validation set, or to the test set, but never to both simultaneously. After this strict separation, the sessions are windowed into fixed sequences after separation, and then resampling is applied to ensure a balance between classes, particularly between normal and malicious traffic. This approach ensures that the test set remains clean, allowing for reliable measurement of the model’s ability to generalise to new scenarios and IoT devices never seen during training.

The AE-LSTM + RL Multi-agent offers both the power of unsupervised reconstruction by detecting anomalies through reconstruction errors, temporal modelling with LSTM, which is a double advantage over zero-day attacks and specific attacks, and distributed, adaptive, and cooperative decision-making thanks to multi-agent RL.

The autoencoder (AE) learns a compact latent representation of normal traffic by reducing dimensions and filtering noise. This latent compression allows anomalies to be detected effectively by comparing the input with its reconstruction. LSTMs are capable of memorising long-term dependencies and managing the gradient effect, making them suitable for detecting anomalies in IoT flows.

Once the features have been extracted by the AE-LSTM, decision-making is entrusted to a multi-agent reinforcement system. Each agent corresponds to an IoT node or IoT traffic source and shares a common PPO policy, but each has its own local observations, learning to cooperate via a centralized reward.

This model is suitable for smart IoT environments because it allows actions (block, alert, ignore) to be dynamically adapted, optimises the overall network strategy (minimising false positives, maximising attack coverage), and coordinates multiple IoT entities for distributed cybersecurity.

It is suitable for massive and distributed IoT environments, detects unprecedented anomalies (zero-day attacks) while the RPs learn to adapt their responses.

The encoder is a 3-layer LSTM with a hidden layer size of 64 units. It transforms each input sequence into a compact latent vector representing the essential characteristics of network traffic. Only the last hidden layer (h_n [−1]) is used as the final representation. The latent vector x_enc = h_[−1] corresponds to the last hidden layer of the LSTM encoder. It is replicated at each time step to form the decoder input. This vector encodes the global summary of the sequence. Decoder (LSTM): It attempts to reconstruct the original sequence from the summary obtained. If the reconstruction is poor (using a measure called MSE), this may indicate an anomaly. The decoder is also a 3-layer LSTM, but inverted: it takes the latent vector (replicated) as input and attempts to reconstruct the original sequence. It measures the model’s ability to memorise and reconstruct traffic behavior. The reconstruction loss is calculated using the MSE-Loss (Mean Squared Error) function. It measures the difference between the original sequence and the sequence reconstructed by the decoder. A faithful reconstruction means that the sequence is ‘normal’, while a high difference may indicate an anomaly or an attack.

RL Multi-agent is based on the CTDE (Centralised Training, Decentralised Execution) paradigm: All agents share a common PPO policy and are trained with a global reward to stabilise optimisation. Each agent observes a different portion of the traffic (local flows, metrics specific to its context) and applies a policy to its own local observation, enabling distributed detection in the IoT environment. Even if the policy is identical, local observations differ, generating heterogeneous behaviours depending on the case.

AE-LSTM + Multi-Agent RL Ablation: The model is trained using the Adam optimiser with the following parameters:

Comparisons: In order to evaluate the relevance of using PPO for intrusion detection in IoT environments, we compare our approach with several representative alternatives: (a) autoencoder (AE) coupled with a simple classifier, (b) supervised sequential models (LSTM, GRU, Transformer), (c) PPO agent alone or combined with an encoder, and (d) single-agent or multi-agent variants.

(a) Autoencoder (AE) + Simple Classifier: The AE extracts latent representations (bottlenecks) from IoT environments, and the classifier is responsible for using them to predict attacks. This configuration allows for effective dimensionality reduction and limits the risk of overfitting, while remaining relatively explainable. However, it relies heavily on the quality of the representations learned by the AE and may lack the ability to capture complex temporal dependencies in network flows.

(b) Supervised Sequential Models: LSTM, GRU, or Transformer networks are trained directly to classify sequences as normal or malicious. They are particularly well-suited for modelling long-term temporal dependencies and perform excellently on rich datasets such as IoT-23. However, they require a large amount of labelled data and are sensitive to zero-day attacks, which are not represented in the training data. Furthermore, their high capacity exposes them to the risk of overfitting in low-data-volume contexts and requires careful encoding and often full supervision.

(c) Reinforcement Learning with PPO: The use of PPO reformulates detection as a sequential decision-making problem, where the agent learns a reward-guided classification policy. This flexibility makes it possible to minimise the false positive rate or give greater weight to false negatives. Without an encoder, the agent learns directly from the normalised features, which simplifies the pipeline but can be inefficient in large state spaces. With an AE, latent representations enhance robustness but at the cost of increased complexity.

(d) Single Agent vs. Multi-Agent: In IoT environments, multi-agent architectures are often preferred for their ability to react locally while sharing global signals. Multi-agent systems can provide increased robustness through majority voting for agents that share the same policy. A single PPO agent is a simple and effective configuration, while a multi-agent approach aims to increase robustness through policy diversity or a majority voting mechanism. Centralised training with decentralised execution (CTDE) is an interesting variant, where multiple agents share a global reward during training but then make decisions independently. However, the multi-agent configuration involves additional complexity, and if the agents share the same policy, the gain over a single agent may remain limited.

Experimentation scripts: The entire process is organized into modular Python scripts:

1) preprocess.data.py: processing, transformation, and export of sequences.

2) train_ae_lstm.py: training of the AE-LSTM (latent, dropout, epochs, batch-size, learning_rate).

3) extract_latent.py: normalised windows in the AE and saving of latent vectors for each sequence.

4) train_ppo_ma.py: takes the latent vector and hyperparameters as input and outputs the class-weighted rewards.

5) evaluate.py: calculates metrics on test data. Compares AE-LSTM + MAPPO and AE-LSTM alone.

6) run_experiments.sh: saves results in a CSV file.

In IoT datasets, normal classes are often much more frequent than attack classes, resulting in significant imbalance. To assess the impact of this imbalance, we test each experimental configuration with and without class rebalancing. Rebalancing is achieved either by oversampling minority classes or by weighting the loss function to give more weight to rare attacks.

Model evaluation is performed along two complementary axes. First, a binary analysis distinguishes normal sequences from all attacks grouped into a single ‘attack’ class. The metrics calculated include overall accuracy, as well as precision, recall, and F1-score for each binary category (normal/attack). This evaluation provides an overview of the model’s ability to detect anomalies.

Secondly, we perform a detailed evaluation by attack type, considering each family of attacks (DoS, Scan, Injection, Malware) separately. For each type, we report precision, recall, and F1-score to identify which attacks are successfully detected and which attacks remain difficult to identify. This dual analysis quantifies the effect of rebalancing on the detection of rare attacks and provides a detailed measure of the model’s performance across all attack scenarios.

Figure 2 illustrates the different stages of the proposed process for intelligent intrusion management in IoT environments. The system relies on a combination of the power of sequential unsupervised learning (AE-LSTM) to model IoT traffic and anomaly detection, with the flexibility of Multi-Agent RL to make distributed and cooperative decisions. The LSTM autoencoder is used for feature extraction, and a set of PPO (Proximal Policy Optimisation) agents act in a multi-agent reinforcement learning environment. This innovative architecture is designed to learn optimal behaviors in response to different types of IoT traffic, based on the compact representations provided by the AE-LSTM model. It meets both the need for robustness in the face of evolving attacks and the requirement for distributed deployment inherent in IoT environments.

As in the previous phase, the raw data from the IoT-23 dataset are first cleaned, normalized, and converted into time sequences. This allows us to capture the evolving behavior of network flows.

The core of the system is based on an LSTM autoencoder, which transforms each sequence into a compact latent vector.

1) Encoding: Two LSTM layers are used to extract a condensed representation (latent vector) from the input sequences. Only the last hidden layer hn [−1] is kept as the final output.

2) Decoding: The latent vector is replicated over time and sent to the LSTM decoder to attempt to reconstruct the original sequence. The reconstruction quality is evaluated using a Mean Squared Error (MSE) loss function.

Once the EA model is trained, the latent vectors from the encoder (the representations (h_n [−1]) are extracted. These vectors contain the essence of the network sequences and constitute the main input to the reinforcement system.

The latent vectors are injected into a custom Gym environment called MultiIoT23Env, which simulates an IoT network with different situations (normal traffic, various attacks). Each agent is responsible for analyzing these representations and deciding whether it is malicious behavior or not, or detecting the type of attack.

Characteristics of a Centralized Multi-Agent System: A centralized multi-agent system is distinguished by the following characteristics, which apply to your implementation:

1) Centralized control: A central entity (here, the MultiIoT23Env environment and the trained PPO model) orchestrates the agents’ actions. Decisions are made based on a global view of the data (latent vectors and labels). Shared observation: All agents receive the same input data (the latents extracted by the AE-LSTM), which contrasts with a distributed system where each agent might have a specific local observation. Unified reward: The reward is calculated based on a common ground truth (true_label), and bonuses are applied globally at the end of the episode, rather than independently for each agent. Single model: A single PPO model is trained and used for all agents, which centralizes learning, unlike a distributed system where each agent might have its own model. Implicit coordination: Decision-making is coordinated by the environment’s central logic, which adjusts rewards and terminates the episode when global conditions are met.

In this environment, several PPO agents are deployed:

1) Precision

$Precision=\frac{VP}{VP+FP}$

2) Recall

$Recall=\frac{VP}{VP+FN}$

3) F1-score

$\text{F1-score}=\frac{2\times Precision\times Recall}{Precision+Recall}$

In order to make PPO agent training clear and reproducible, we define a simple reward function and a well-defined episode structure. Each episode corresponds to a time window of size T = 60 time steps, from the network stream. At each step of the episode, the agent receives the current state (vector of raw features or latent representation extracted by the AE) and predicts whether the sequence is normal or corresponds to an attack.

The agent intuitively receives a positive reward (+1) when it correctly predicts the state of the sequence and a negative reward (−1) when it is wrong. The reward can be weighted to compensate for class imbalance and give more importance to rare attacks. This simplicity facilitates PPO convergence and allows the agent to quickly understand the expected behaviour.

Each episode ends when the sequence of length T is completely exhausted. The cumulative reward per episode is recorded in order to easily track the evolution of the agent’s performance. This approach guarantees a standardized structure for training, facilitates comparison between different configurations (single agent vs. multi-agents, with or without AE, class rebalancing), and improves the reproducibility of experiments.

The action consists of predicting the class of the current sequence: 0 = normal, 1 = attack (or multi-class depending on the type of attack). This allows the immediate reward for this step to be determined. Repeat the experiments on at least 3 different random seeds to verify stability using 95% confidence intervals ( Figure 3 ).