Historically, optimization theory was introduced by Kantorovich in 1960, who advocated obtaining an optimum solution to a presented problem by applying a definite, scientifically based method. Optimization is a discipline that studies how actions are chosen to achieve objectives optimally [1] . Optimization delineates the methodology or process of improving a system’s or network’s efficacy or functionality [2] . Optimization generally entails the modification of variables to maximize or minimize a specific objective function while simultaneously balancing the constraints of the problem. DL is a specialized branch of ML that harnesses neural networks to autonomously analyze datasets, allowing computer systems to achieve refined performance through experience and continuous data exposure [3] . DL algorithms outperform traditional ML in cyberattack detection, leveraging their multi-layered structure and powerful extraction techniques to efficiently derive meaningful insights from training data [4] . Challenges with most deep learning algorithms include lengthy training periods, manual parameter optimization, and accuracy deficiencies in detection [4] [5] .

Optimization is crucial in refining deep learning algorithms for improved efficiency and reliability for optimal performance. In cybersecurity, optimization techniques ensure that detection models can accurately identify threats while reducing false positives and computational overheads [6] . Optimization is valuable not because it seeks to understand every system detail, but because it identifies the most efficient way to refine and adjust it with minimal effort [7] . Optimization, in its simplest form, is about selecting inputs that will result in the best possible outputs of a given system. Optimization techniques help analyze the operational space and forecast the necessary adjustments to system parameters, ensuring peak performance [7] . The choice between stochastic and deterministic optimization methods relies on factors such as the demand for computational efficiency, the complexity of a problem, and the precision of available system data [8] .

In stochastic optimization, the techniques applied lack predictability in the search process, such as through approximations. In contrast, deterministic optimization methods follow a fixed, predictable path to find a solution, relying on precise calculations of gradients or other problem-specific derivatives [9] . This integration of randomness generalizes stochastic optimization, as randomness can be incorporated using various mechanisms [8] . There are different classes of stochastic optimization algorithms, as indicated in the following.

1) Simulated Annealing (SA): This is a probabilistic technique modeled according to the annealing process in metallurgy, symbolizing the physical process of heating and cooling metals [10] . After identifying a search space, the SA algorithm comprehensively explores different combinations to find an optimal solution.

2) Particle Swarm Optimization (PSO): This draws inspiration from the collective dynamics of natural swarms, such as the flocking of birds, where the movement of particles is simulated within a search space to find an optimal solution [11] [12] .

3) Game Theory-based optimization (GT): GT-based methods seek to simulate the dynamics of a game with players pursuing individual goals, grounded in the principles of Nash’s game theory [13] [14] .

4) Evolutionary Algorithms (EA): EAs are techniques inspired by the replication of the evolutionary process as articulated in Darwin’s theory of natural selection [15] [16] . Genetic Algorithms (GA) are a subset of EA that mimic natural evolution to achieve optimal solutions to complex problems through successive iterations of multiple potential solutions [17] .

A key benefit of stochastic optimization is its ability to navigate beyond local minima and thoroughly explore the design space due to its inherent randomness [18] . The challenge is reconciling the comprehensive exploration of the solution space with the timely convergence to an optimal solution [19] .

Algorithm tuning involves continuous adjustment of various control parameters [20] , significantly influencing an algorithm’s performance [21] . Stochastic optimization algorithms mainly draw inspiration from natural processes and adaptations to solve optimization [22] . The formulation of a specific optimization problem and its environment influence the performance of the applied optimization technique [23] . Stochastic optimization techniques generally offer advanced methods for solving optimization problems [24] .

The optimization techniques proposed in our framework are well-grounded practices. However, their strategic integration for cyberattack detection, by aligning architectural design, training procedures, and tuning dynamics, presents a unified approach to cybersecurity modeling. In contrast to previous studies (discussed in Section 2), which concentrate exclusively on enhancing a single phase of the model life cycle or utilize generic parameter optimization, the framework in this study establishes a unified, mathematically defined optimization pipeline that fluidly incorporates feature selection, hyperparameter optimization, gradient-based training, and regularization for the CI-CIDS-2017 dataset and web-based attack profiles.

The justification for this integrated methodology is to address practical constraints related to class imbalance, convergence efficiency, and real-time detection rates in high-traffic scenarios. Comparative experiments (section 6.3) demonstrate that the fully integrated approach achieves a false positive rate of 1.1% and a false negative rate of 5.3%, representing measurable enhancements over a baseline LSTM model lacking staged optimization, which recorded false positive and false negative rates of 3.4% and 8.9%, respectively.

This paper is structured as follows: Section 2 presents the related studies on optimization in cyberattack detection. Section 3 discusses the general optimization problem modeling task. Section 4 highlights the mathematical modeling of the cyberattack detection task as an optimization problem. Section 5 draws attention to the experimental design, while Section 6 discusses the study results. Finally, Section 7 gives the conclusions of the study.

ML has been an optimal area for algorithm optimization, but it has also led to more research in optimization methods [25] [26] . ML focuses on making machines learn from experience. Learning involves using observations to search for an environment’s model and predict new queries for a problem [26] . In the earlier days of ML, there was more emphasis on models catering to specifically identified input data structures due to low computational capabilities and limited data access, which allowed the design of models to specifically leverage data structures and tune them for tasks with small data amounts [27] . The merit was that it facilitated the models’ comprehensive theoretical analysis and guaranteed the models’ performance [28] .

Using off-the-shelf algorithms to search for optimal models is possible, but new optimization models must be designed specifically for the particular problem, enhancing ML efficiency [29] . Alternatively, with improved computational capabilities and larger amounts of data, it has been possible to utilize more flexible and complex models, such as Deep Neural Networks (DNN). Regardless of complex models requiring more resources and data to fit appropriately to the data and problem, the advantage is that similar models may be adopted without modifying them for specific problems, thus they perform better in real-world problems.

Previous decades have witnessed a shift in how intelligent systems are designed [30] . Rather than mimicking human decision-making, machine design mimics the human learning process. Such a framework involves a model with variables from appropriate data specific to the problem or task. Popular optimization approaches are applicable in addressing the learning problem in the case of ML and DL [31] . Also, the adoption of generic optimization may be inefficient due to the problem’s scale on the loss function and output space. To address a dataset’s large size, optimization methods’ stochastic variants that only work with a subset of the dataset at a time have been popularized [8] . Further, dealing with complexity in loss functions and output space associated with the learning problem calls for developing methods that leverage the structure respective to the task to enhance the optimization’s efficiency. ML optimization encompasses a broader range of techniques, such as neural networks, support vector machines (SVM), linear and logistic regression [32] .

In the context of cyberattack detection, several studies have utilized optimization, such as Brindha et al. [33] , who applied optimized deep learning approaches; Do et al. [34] , who studied optimization using ML; Injadat et al. [32] , who researched a multi-stage optimized ML framework; Nayak et al. [35] , who studied an ML and Bayesian optimization-driven intelligent framework for Internet of Medical Things (IoMT) cyberattack detection; and Zhang et al. [36] , who focused on low-rate DoS attack detection using Power Spectral Density (PSD)-based entropy and ML. However, most of these works either lack a unified optimization framework or do not fully address practical constraints concerning hyper-parameter tuning, class imbalance, and real-time attack detection. This study, therefore, aims to focus on these challenges by considering the optimization lifecycle of cyberattack detection under practical constraints.

The following discussion provides a general definition of the optimization problem. In a task requiring predicting a $y$ variable that depends on $x$ input, the application of ML essentially involves two computational problems. In the assumption as presented in Equation (1), that:

$y=f\left(x;\theta \right)$ (1)

where the $f$ function maps input to output and has $\theta$ as the parameters, then the initial computation problem is the computation of the $y$ output by evaluation of $f\left(x;\theta \right)$ at $x$ input, which is called the inference or prediction problem.

Secondly, if given data $D$, parameters $\theta$ of function $f$ may be estimated in Equation (2) as:

${\theta }^{*}=\arg {\min }_{\theta }L\left(f,D\right)$(2)

$L$ is the loss function measuring the parameter configuration quality, which is called the learning problem. In general, the learning and inference problems may be seen as optimization problems in ML and DL. The inference problem involves choosing an output $y$ from a set $Y$, maximizing a score estimating the prediction’s quality as in Equation (3):

$y^{*}=\underset{y\in Y}{\arg \min }\text{score}\left(x,y\right)$ (3)

The $Y$ domain differs for each task. For a classification problem, the set would be categorical, but for a regression task, it would be a continuous space. The domain set may be sophisticated, such as the possible rankings of a set in ranked retrieval tasks. Whereas deploying off-the-shelf optimization models to solve inference problems is possible, other cases require the development of new optimization models to leverage the problem’s structure to solve it effectively.

For the learning problem in a standard setting, according to Equation (4), parameters $\theta$ of the model $f\left(x;\theta \right)$ are selected since the loss function’s expected value $L\left(f\left(x;\theta \right),y\right)$ is minimized by the model over the data distribution $P_{data}$.

${\theta }^{*}=\underset{\theta }{\arg \min }{E}_{\left(x,y\right)~P_{data}}\left[L\left(f\left(x;\theta \right),y\right)\right]$ (4)

The risk is the objective of the optimization problem [28] , and in computing, this requires knowledge of the data distribution $P_{data}$, which is generally not known. Hence, an empirical estimate is optimized as the loss function expectation on the training data $D_{train}$ known as empirical risk, as in Equation (5).

${\theta }^{*}=\underset{\theta }{\arg \min }\frac{1}{\left|D_{train}\right|}{\displaystyle \underset{\left(x,y\right)\in D_{train}}{\sum }L\left(f\left(x;\theta \right),y\right)}$ (5)

Whereas the training set is only used for model training, it is surmised that it performs efficiently on unseen data in the selected distribution. A model’s overall performance may be evaluated using the testing data, $D_{test}$, to determine risk. A learning algorithm that overfits the training set model will result in negative overall performance. However, overfitting can be addressed by adding additional regularization to the equation’s objective function.

The relationship between an independent and dependent variable, $x$ and $y$, respectively, such as in linear regression, with $n$ observations in a given dataset, $\left\{\left(x_1,y_1\right),\left(x_2,y_2\right),\cdots ,\left(x_n,y_n\right)\right\}$, aims to identify the parameters $w$ as in Equation (6) where:

$\left(w,x_i\right)=y_i,\forall i$ (6)

The inner product of $w$ and $x$ is represented by $\left(w,x\right)$. If the relationship between x and $y$ is non-linear, then the problem may be unsolvable. This can be avoided by getting the parameters of $w$ as close as possible to $y$ (average), as in Equation (7):

${\min }_{w}f\left(w\right)=\frac{1}{n}{\displaystyle {\sum }_{i=1}^n{\left(\left(w,x_i\right)-y_i\right)}^2}$ (7)

Designing ML tasks as optimization problems can be done, as in Equation (8), in the following form:

${\min }_{w}f\left(w\right)=\frac{1}{n}{\displaystyle \underset{i-1}{\overset{n}{\sum }}{f}_i}\left(w;x_i\right)$ (8)

where $\left\{x_i\right\}$ is data from an unknown probability distribution $p$. In ML and DL systems, each term $f_i\left(w;x_i\right)$ shows how a model with parameters $w$ fits a particular observation $x_i$. With a dataset $D_a$ having $n$ samples $\left\{x_i\right\}$, $f\left(w\right)$ identifies how well the model fits the entire corpus of data on average. This is an empirical risk minimization problem, which is an estimate of the true problem to be solved, i.e., the expected risk minimization problem: ${\min }_w{\mathbb{E}}_{x_i~p}\left[f_i\left(w;x_i\right)\right]$. Generally, without information on the distribution $p$ to solve a risk minimization problem, we can solve Equation (8) alternatively.

If the objective is to make a prediction based on an input, each data sample $x$ in the dataset $D_a$ has a corresponding label $y=C\left(x\right)$, for an unknown labeling function $C$. In this case, a training pair refers to the tuple $\left(x,y\right)$, and $x$ is considered to be a $d$-dimensional vector with $x\in {ℝ}^d$, unless specified otherwise.

ML and DL models use objective functions formulated in Equation (8). Logistic regression, which is useful for binary classification, uses the objective function as in Equation (9):

$f_i\left(w\right)=\text{log}\left(1+\exp \left(-y_i\left(w,x_i\right)\right)\right)$ (9)

where $y_i$ represents, across $n$ observations, an averaged binary label (±1).

The framing of cyberattack detection systems as optimization problems targeting accuracy, efficiency, and robustness is necessary for optimal performance. Hence, by formulating the detection task as an optimization problem, the subsequent mathematical modelling enables the development of a structured and systematic framework for model development. The following is the definition of the optimization problem.

This section provides a mathematical definition of the cyberattack detection optimization problem. It begins by first defining key terms and variables that will be used in formulating the cyberattack optimization problem, before giving a formal definition of the problem.

Mathematical optimization techniques are used to find optimal values for the deep learning-based model parameters, P.

The results of the experimental work follow.

This paper presents a comprehensive optimization-driven framework for cyberattack detection, offering extensible theoretical and practical insights for cybersecurity modelling and a foundation for future research and actual deployment in security environments. The framework transforms standard optimization components into a domain-specific, DL-based system, validated across multiple threat scenarios in the CIC-IDS 2017 dataset, offering an intelligent solution for cyber defense. Deploying the optimization framework across architectural design, training techniques, and hyper-parameter tuning highlights its efficiency and robustness in model development. Extensions of this framework to other detection systems and the application of ensemble and reinforcement learning for dynamic defense strategies lay the groundwork for future research.

Conceptualization, G.A.O. and N.M.; methodology, G.A.O. and C.Y.; software, G.A.O. and C.Y.; validation, G.A.O., N.M. and C.Y.; formal analysis, G.A.O. and N.M.; investigation, G.A.O. and N.M.; resources, G.A.O.; data curation, G.A.O. and C.Y.; writing—original draft preparation, G.A.O.; writing—review and editing, G.A.O., N.M. and C.Y.; visualization, G.A.O. and C.Y.; supervision, N.M. and C.Y.; project administration, N.M. and C.Y. All authors have read and agreed to the published version of the manuscript.

The CIC-IDS 2017 dataset used in the research is available at: https://www.kaggle.com/datasets/amanverma1999/a-comprehensive-dataset-for-ddos-attack .

We thank the members of the School of Informatics and Innovative Systems at Jaramogi Oginga Odinga University of Science and Technology for their valuable insights during the research.

The following abbreviations are used in this manuscript: