Anomaly detection has been a critical component of cybersecurity research, particularly for identifying unusual patterns in network traffic or user behavior that may indicate potential threats [14] . Traditional methods, such as clustering and statistical models, have been widely employed due to their simplicity and interpretability [15] . However, these approaches often struggle to handle high-dimensional data and adapt to the dynamic and evolving nature of modern cyberattacks [1] . Recent advancements in machine learning, particularly autoencoders, have shown significant promise in modeling normal behavior and detecting anomalies by measuring reconstruction errors [2] . These methods enable systems to learn the baseline patterns of normal operations and flag deviations as potential threats [3] . Despite their effectiveness, challenges remain, particularly in handling imbalanced datasets where anomalies represent a small fraction of the data and in adapting to highly variable real-world conditions [4] . These limitations highlight the need for robust preprocessing techniques and advanced algorithms to improve the reliability and scalability of anomaly detection systems [5] .

Fraud detection remains a major focus in the financial sector, with numerous machine learning models applied to classify transactions as fraudulent or legitimate. Logistic regression, decision trees, and ensemble methods like random forests have been commonly used, relying on features such as transaction history, geographic patterns, and user behavior. Despite their effectiveness, these models often struggle to detect emerging fraud techniques and highly sophisticated attacks. Adaptive learning approaches, such as those incorporating real-time feedback, have been introduced to address these limitations. However, class imbalance—where fraudulent transactions represent a small fraction of the data—continues to pose significant challenges, often requiring advanced preprocessing techniques like oversampling or cost-sensitive learning [15] .

Distributed Denial of Service (DDoS) attacks pose significant risks to online banking systems, often overwhelming servers and causing service outages. Traditional mitigation strategies involve static rule-based systems for resource allocation, which lack the flexibility to respond dynamically to varying attack intensities. Reinforcement learning techniques, particularly Q-learning, have shown promise by enabling systems to dynamically optimize server resource allocation based on network conditions. By learning from real-time traffic patterns, Q-learning reduces latency and ensures system availability even during high-traffic scenarios. However, the application of reinforcement learning for DDoS mitigation in banking environments is still relatively unexplored.

While significant progress has been made in anomaly detection, fraud detection, and DDoS mitigation individually, the lack of unified frameworks addressing these challenges simultaneously limits their practical applicability. Most existing studies focus on isolated solutions, neglecting the interconnected nature of cyber threats in modern banking systems. By integrating techniques like autoencoders, logistic regression, and Q-learning into a cohesive system, this study aims to address multiple threats simultaneously, providing a scalable and adaptive solution for real-time cybersecurity in consumer banking websites [2] .

The data set used in this study comprises a combination of real-world transaction logs, user activity metrics, and network traffic data, along with synthetic attack scenarios to represent rare but critical events [4] . The real-world data consists of approximately 500,000 records, with 60% labeled as legitimate transactions, 30% flagged as suspicious, and 10% confirmed as fraudulent [5] . Key attributes include transaction amounts, user activity patterns, IP addresses, timestamps, and geographic locations [6] . Synthetic data was added to simulate specific attack behaviors, such as sudden traffic spikes for DDoS scenarios and unusually high-value transactions for fraud detection [7] . Approximately 5% of the dataset contained missing values, which were imputed using mean imputation for numerical features and mode imputation.

Anomaly detection is performed using autoencoders, which are trained to reconstruct normal patterns in user behavior and network traffic. A significant reconstruction error indicates the presence of an anomaly. The loss function used to train the autoencoder is given as [4]

$E=\left(\frac{1}{n}\right)\Sigma {\left(x_i-{\hat{x}}_i\right)}^2$ (1)

Fraud detection leverages logistic regression to classify transactions as legitimate or fraudulent based on extracted features. The logistic function predicts the probability of fraud as [5]

$P\left(y\right)=\frac{1}{1+e^{-\left({\beta }_0+{\beta }_{\left(1x_1\right)}+\cdots +{\beta }_{\left(n{x}_n\right)}\right)}}$ (2)

To mitigate Distributed Denial of Service (DDoS) attacks, reinforcement learning via Q-learning is employed to optimize server resource allocation. The Q-value for a given state-action pair is updated as [6]

$Q\left(s,a\right)\leftarrow Q\left(s,a\right)+\alpha \left(r+\gamma \backslash {\max }_{a}Q\left(s^{\prime },a\right)-Q\left(s,a\right)\right)$ (3)

The proposed anomaly detection module, based on autoencoders, was evaluated using the AUC (Area Under the Curve) metric across multiple experimental runs. The model achieved consistent performance, with an average AUC of 0.93, demonstrating its robustness in identifying deviations from normal patterns in network traffic. Figure 1 illustrates the AUC scores across different runs, showing steady improvements due to hyperparameter tuning and optimized reconstruction error thresholds.

The fraud classification component utilizes logistic regression to classify transactions as legitimate or fraudulent. Precision, recall, and F1-score metrics were used to evaluate the model’s performance, with values of 92%, 90%, and 91%, respectively. The use of the SMOTE algorithm for handling class imbalance significantly enhanced the detection of fraudulent transactions. Figure 2 presents the comparison of these metrics, highlighting the effectiveness of the model in fraud detection.

The Q-learning-based DDoS mitigation module dynamically allocated server resources to minimize response times and improve resource utilization. The model demonstrated a 35% reduction in server response times under extreme traffic conditions, as shown in Figure 3 .

Fraudulent transactions represented less than 10% of the dataset, causing significant class imbalance issues. This imbalance resulted in biased predictions, where the model favored the majority class (legitimate transactions) over the minority class (fraudulent transactions). To address this issue, the Synthetic Minority Oversampling Technique (SMOTE) was applied to oversample the minority class. This method created synthetic samples to balance the dataset, improving the model’s ability to detect fraud effectively. Additionally, cost-sensitive learning was explored to prioritize minority class predictions without sacrificing overall accuracy.

The network traffic dataset contained numerous irrelevant features, leading to computational inefficiency and increased complexity in anomaly detection. High-dimensional data can obscure meaningful patterns, reducing the performance of machine learning models. To overcome this, dimensionality reduction techniques, such as autoencoder-based feature extraction, were employed. These methods identified and retained critical features, enhancing the accuracy and efficiency of the anomaly detection module while reducing computational overhead.

Processing high volumes of transactional and network traffic data in real time posed significant computational challenges. Traditional methods were unable to meet the low-latency requirements for real-time anomaly detection and fraud classification. To address this, optimized algorithms and GPU-accelerated computing were implemented. These solutions reduced response times and ensured that the system could handle dynamic environments effectively.

Tuning the hyperparameters for the Q-learning model, such as the learning rate $\alpha$ and discount factor $\gamma$, was time-consuming and computationally intensive. These parameters significantly impacted the model’s performance and stability. To streamline the process, automated grid search and cross-validation techniques were used to identify optimal hyperparameter values, minimizing manual experimentation while maximizing model efficiency.

The DDoS mitigation module encountered challenges in adapting to sudden spikes in network traffic. Traditional static resource allocation methods were insufficient to address dynamic traffic conditions. The Q-learning model was designed to adapt resource allocation dynamically based on real-time feedback, ensuring optimal server utilization and performance even under extreme traffic conditions.