SAEAs are a framework for evolutionary algorithms that has garnered significant attention in recent years and has become a research hotspot for Expensive Multiobjective Optimization Problems (EMOPs) [8]. Commonly used surrogate models include:

1) Gaussian Process Regression (GPR) [9]: It models the probability distribution of the objective function through kernel functions, with the output mean and covariance given by Equation (1).

where k^T is the training data covariance matrix, and k(x, X) is the kernel function vector between the test point x and the training set X.

2) Random Forest (RF) [10]: Random Forest predicts results through decision trees, with its output being the mean of all trees as shown in Equation (2).

where T_i is the predicted value of the i-th tree, and M is the number of trees.

3) Support Vector Regression (SVR) [11]: SVR is trained by maximizing the margin and minimizing prediction errors, with its objective function given by Equation (3).

where ϕ ( x i ) is the kernel mapping function, ε is the tolerance error, and ξ is the slack variable.

To compare the tendencies and limitations of different heterogeneous surrogate models, based on Table 1, this paper statistically summarizes the tendency characteristics of Gaussian Process Regression, Random Forest, Support Vector Regression, etc., in terms of problem dimensionality, smoothness, complexity, etc., as listed in Table 1.

Table 1. Applicability analysis of heterogeneous surrogate models.

The overall workflow of the proposed adaptive surrogate model selection framework based on the SISSO model selector is shown in Figure 1.

Step 1: Determine basic function characteristics. Select 15 representative basic functions (F1 - F15) from the optimization function library, covering different types such as unimodal, multimodal, hybrid, and composite. For each basic function, perform uniform random sampling, with the sample size gradually increasing from 10 to 200. For the sampled data under each sample size, compute eight feature values: mean of objective values, standard deviation, skewness and kurtosis, correlation strength, nonlinear index, smoothness index, and ruggedness index. Observe the trend of each feature value for each basic function as the sample size increases, plot the feature-value vs. sample-size change curve, and analyze the convergence of the feature values. The feature value at convergence is taken as the function’s characteristic value.

Figure 1. Overall workflow of adaptive surrogate model selection framework based on sisso model selector.

Step 2: Determine the optimal model for basic functions. First, select 12 candidate surrogate models (M1 - M12). For each basic function, use the RandomizedSearchCV() method from the sklearn library to optimize model parameters within a defined hyperparameter grid, determining the optimal parameters for each model fitting each function. Using the optimized surrogate models, perform five-fold [cross-validation] {.mark} on the 15 basic functions, and count the distribution of core parameters where each model performs best across all functions. Under the optimal parameter configuration, re-evaluate the performance of each model under different sample sizes. The model that does not change when the sample size increases sufficiently is considered the optimal model for the basic function.

Step 3: Implementation of the SISSO model selector. Evaluate the surrogate models’ fitting effectiveness on test functions using the R² metric, obtaining an R² performance matrix for each model on the basic functions. Based on R² values, calculate the Euclidean distance between models to construct a model correlation matrix. Build a complete graph with models as nodes and Euclidean distances as edge weights. First, transform the model numbering problem into a Traveling Salesman Problem to find the shortest traversal path. Number the models according to their order on the shortest path. Use the SISSO algorithm to obtain a mapping function from the 8 features to 5 model numbers, implementing model selection.

3.2.1. Initial Sampling

Use Symmetric Latin Hypercube Design (SLHD) to generate initial sample points [12].

where, x⁽ⁿ⁾ is the n-th sample point, l is the lower bound of variables, u is the upper bound of variables, and d is the problem dimensionality.

3.2.2. Feature Extraction Technique

Extract multi-dimensional features from the problem and sampled data. Problem structural features: dimensionality, search space volume, boundary range. Data statistical features: mean of objective values, standard deviation, skewness, kurtosis. Function characteristic features: correlation strength, nonlinear index, smoothness index, ruggedness index [13][14].

For a given optimization problem, let the variable dimensionality be d, the lower variable bound be l, the upper bound be u, the initial sample points be X, and the objective values be y. Feature extraction can be uniformly expressed as:

where m is the feature vector, and ϕ ( • ) represents the feature extraction operator.

① Problem structural features. Include dimensionality, space volume, boundary range. Define the problem structural feature vector as:

where space volume and boundary range can be expressed as:

② Sampled data statistical features. Include mean of objective values f1, standard deviation f2, skewness f3, kurtosis f4. Define the data statistical feature vector as:

where the skewness coefficient and kurtosis coefficient are respectively:

③ Function characteristic features. Include correlation strength f5, nonlinear index f6, smoothness index f7. Define the function characteristic feature vector as:

Correlation strength (ρ): This feature measures the degree of linear correlation between input variables, with the formula as follows.

where, c o r r ( X i , X j ) is the Pearson correlation coefficient between the i-th and j-th dimension variables, given by:

Here, n is the sample size, x ¯ i and x ¯ j are the sample means of thei-th and j-th dimensions, respectively.

Nonlinear index (η): This feature measures the degree of nonlinearity of the objective function, obtained by comparing the mean squared error (MSE) of linear and nonlinear models.

Here, MSE_linear is the MSE of a linear model on the samples, and MSE_nonlinear is the MSE of a nonlinear model on the samples. Typically, a linear model and a nonlinear model are trained on the sample data, their MSE on the training samples is calculated, and then the nonlinear index η is computed. If the result is negative, it is set to 0.

Smoothness index (ξ): This feature measures the smoothness of the objective function in the input space, obtained by calculating the rate of change of the objective function value between sample points.

Here, N p = ( n 2 ) = n ( n − 1 ) 2 is the number of sample point pairs (i.e., all pairwise

combinations), y_i and y_j are the objective function values of sample points X_i and X_j respectively, ‖ x i − x j ‖ is the Euclidean distance between sample points, i.e.:

Ruggedness index (r): The ruggedness index f8 is an indicator used to quantify the complexity or irregularity of the local structure of an optimization problem’s objective function surface. It reflects how “sharp” or “chaotic” the function’s variation is within a small region.

Here λ i ( k ) represents the local eigenvalues calculated for the k-th sampling point or within a certain sub-region, Var denotes variance, and Mean denotes the mean.

Test functions have fitness landscapes with different characteristics, which also leads to inconsistent performance of a single model fitting different test functions. Analyzing the characteristics of basic functions and determining the most suitable surrogate models is the foundation for adaptively selecting surrogate models based on optimization problem features. The basic functions used in this paper are listed in Table 2, numbered F1 - F15 sequentially from the Zakharov function to the Griewank function.

Table 2. Basic function information table.

To determine the characteristic value of each basic function, statistical experiments were conducted. As the sample size increases, the function feature values tend to stabilize, and the value at this point is taken as the function’s characteristic value. Research found that when the sample size reaches 200, the function value tends to stabilize. Statistics in Table 3 show the feature values of basic functions when the random sample size is 200.

The experimental scheme for determining optimal parameters adopted a random search method to identify optimal parameters for multiple machine learning models. Random search randomly selects parameter combinations within a defined parameter grid and evaluates model performance via cross-validation, using the RandomizedSearchCV method from the sklearn.model_selection package. This method is more efficient than grid search because it does not evaluate all possible parameter combinations but randomly selects combinations within a specified number of iterations, aiming to find good parameter configurations in a shorter time. Each model performed corresponding parameter search and best parameter selection for the 15 functions. The core parameter distribution refers to the most frequently occurring value for each individual parameter across all functions, and the optimal parameter combination refers to the most frequently occurring single set of parameters. This combination is used as a reference for subsequent experiments. Table 4 provides the optimal parameters for each model, which are used in all subsequent experiments.

Table 3. Feature values of basic functions with random sample size of 200.

Table 4. Optimal parameter table for each model.

Based on the determined optimal model parameters, statistical experiments were conducted to determine the optimal model for each function under different sample size conditions. Both training and test sets were generated using uniform random sampling, with the test set remaining consistent after generation. Table 5 lists the optimal models for basic functions as the sample size changes. The table shows that the optimal model stabilizes when the experimental sample size reaches 200.

Table 5. Optimal models for basic functions.

To ensure that similar models are encoded with closer numbers, this paper designs a model similarity metric. Models are trained with 200 training samples, and the R² values of each model fitting the 200 samples in the test set for each basic function are recorded, as shown in Table 6.

From Table 7, it is observed that for function F8, the corresponding test set R² values are all negative because it has many local optima, and all models fit poorly, making it unsuitable as a basic function. Therefore, this paper discards it. Based on the above table, the Euclidean distances between the 12 models are calculated to obtain the model correlation matrix in Table 5.

Table 6. Test set R² values under optimal parameters for different models.

Table 7. Model correlation matrix.

The Euclidean distances between models are treated as an adjacency matrix, where the graph nodes are model types and edges are Euclidean distances. The shortest path traversing all nodes in the graph is found. The longest edge on the path is selected, and numbering starts from one vertex of this edge. In this paper, poorly performing models are discarded, and only the optimal models for the 200-sample point condition are used, i.e., m3, m9, m10, m2, m4. Using brute-force solving for the Traveling Salesman Problem, the shortest path is found to be m3, m4, m2, m10, m9, which are then numbered sequentially as 1, 2, 3, 4, 5. The corresponding models are GPR (Gaussian Process Regression, M3), KernelRidge (Kernel Ridge Regression, M4), GBR (Gradient Boosting Regression, M2), RFR (Random Forest Regression, M10), and MLPRegressor (Multi-Layer Perceptron Regressor, M9). By rotating and translating the basic functions, a library of 165 transformed basic functions is obtained. These functions are sampled with a sample size of 200, and their features and optimal models are recorded, resulting in a table of feature and optimal model information for each basic function (see Table 8). Using the feature values as input and Best_ID as output, the SISSO algorithm is used to train a model, obtaining the model selector shown in Equation (19).

where

Term1 = (−0.11E+01) * (abs((f7/f5) − abs(f6 − f8)) − (f7/f5))

Term2 = (0.64E−02) * ((f1 * f4)/((f4)³ + (f5/f7)))

Term3 = (0.19E+00) * (((f3 − f4) − f8)/abs((f8)⁻¹ − f6))

Constant = (0.25E+01)

Table 8. Basic function characteristics and optimal model information.

Mapping from optimization problem features to optimal model number is achieved by rounding the output of the model selector. The advantage of the SISSO model selector is that it can discover complex nonlinear relationships between features and the optimal model with only a small amount of sample data. Finally, in the evaluation and validation phase, the system performs large-scale experiments on test functions, analyzes the stability of model selection under different sample sizes, determines the most frequently selected model at each sample size through multiple repeated experiments, and records the corresponding best performance metrics.

This framework constructs an optimization problem model selection system based on an SISSO model selector [15]. The core idea is to statistically extract problem features during the optimization process, feed these features into the pre-trained SISSO model selector, and dynamically select the optimal surrogate model for different problems. The entire implementation follows a complete closed-loop process of “data collection—feature extraction—model selection—model-driven optimization”.

First, in the data collection stage, intermediate solutions and their fitness values evaluated by the algorithm in each iteration are recorded to build a training sample database. Then, in the feature extraction stage, eight features of the optimization problem are statistically extracted based on the training sample database: including mean of objective values, standard deviation, skewness, kurtosis, correlation strength, nonlinear index, smoothness index, and ruggedness index. Next, in the model selection stage, these features are input into the pre-trained SISSO model selector. The framework flowchart is shown in Figure 2.

Figure 2. Adaptive surrogate-assisted evolutionary algorithm based on SISSO model selector.

3.7.1. Multi-Subpopulation Selection Strategy

To enhance the algorithm’s global convergence capability and local exploitation efficiency on complex fitness landscapes, this paper proposes a multi-subpopulation selection strategy during the intermediate solution generation phase. Its core idea originates from the Multivariant Optimization Algorithm (MOA) [16]. In this strategy, intelligent search individuals are organized into multiple subpopulations with different responsibilities, dedicated to global exploration and local exploitation, respectively, thereby systematically coordinating the exploration-exploitation balance during the search process. This strategy divides the population into multiple subpopulations, allowing each subpopulation to independently perform sampling and evaluation, followed by explicit task allocation. Some subpopulations are assigned the role of global exploration, continuously searching the entire solution space to reduce the risk of getting trapped in local optima; the remaining subpopulations take on the role of local exploitation, performing fine-grained searches around the current best solution to improve solution quality within potential regions. This series of mechanisms makes it particularly suitable for complex optimization problems with multiple traps, deceptive, or randomly located optima, overall enhancing the probability of global convergence while also strengthening local exploitation capability.

3.7.2. Feature Extraction and Model Selection

After completing subpopulation division and sampling, the system enters the feature extraction and model selection stage. The core goal is to dynamically select the optimal surrogate model based on the characteristics of the current optimization problem, thereby achieving high-precision fitting of complex fitness landscapes. The feature extraction process obtains eight key features from the sampled data of each subpopulation, including objective value mean, standard deviation, skewness, and kurtosis that reflect data distribution characteristics, as well as correlation strength, nonlinear index, smoothness index, and ruggedness index that characterize the function structure. These features collectively describe the global and local properties of the problem. Based on the extracted features, the system employs the SISSO model selector as the classifier. This selector uses symbolic regression to screen out the most discriminative feature combinations from the high-dimensional feature space and constructs a mapping function as shown in Equation (19), outputting a model number corresponding to different surrogate models. During optimization, the system re-extracts features and inputs them into the SISSO model selector in each iteration, selecting the surrogate model most suitable for the current stage in real-time. This enables a flexible response to dynamic changes in the fitness landscape, improving fitting accuracy and optimization efficiency.

The four models—ElasticNet, Lars, Lasso, and Ridge—used as baselines in this paper are all independent optimization algorithms.

During the experimental process, the following parameter settings were found to achieve the best results: problem dimension D = 2, maximum iterations I = 7, population size P = 6, initial sampling points S = 20, subpopulation configuration of 4 subpopulations with 100 samples each, Gaussian sampling, radius scaling 0.7, each function run 30 times. Comparisons were made with four popular surrogate models in recent years: ElasticNet, Lars, Lasso, and Ridge. The symbols “+”, “−”, and “≈” indicate that the proposed SISSO-ASEA algorithm is superior, inferior, or shows no significant difference compared to other algorithms, respectively.

To verify the performance of the SISSO-ASEA algorithm, we integrated the SISSO optimization selector framework, named SISSO-Adaptive-Optimizer (SAO), which incorporates the complete closed loop of “sampling → feature extraction → model selection → optimization search”. This paper selected four models—ElasticNet, Lars, Lasso, and Ridge—for comparison with SAO. Table 9 shows the mean and variance of the error obtained from 30 independent runs of SAO and the other four models in 2 dimensions. The results show that SAO exhibits significant advantages on the vast majority of test functions. Among the nine test functions, SAO outperforms ElasticNet on 7 functions, Lars on 2 functions, Lasso on 7 functions, and Ridge on 7 functions, demonstrating its good generalization ability and stability. Particularly on function f9, SAO’s mean error is far lower than that of the comparison algorithms, indicating its outstanding capability in handling highly nonlinear, multimodal problems. Furthermore, on functions f2 and f4, SAO consistently achieves lower mean errors, showing its strong adaptability and predictive accuracy for different function characteristics. Although on individual functions like f1 and f3, SAO performs comparably to some algorithms, its overall performance is significantly leading, with relatively stable variance control. This further verifies that SAO, as an adaptive surrogate model optimization framework based on the SISSO algorithm, has strong competitiveness and practicality in expensive optimization problems.

Table 9. Comparison results of SAO and 4 models on CEC2022 Test functions in 2 dimensions.

To more intuitively demonstrate the performance of SISSO-ASEA, we compared the convergence of SISSO-ASEA and the other four algorithms in 2 dimensions. The experiment selected f1, f5, f9, and f12 from the CEC2022 test functions. From Figure 3, it can be seen that SISSO-ASEA, compared to the other four surrogate models in later stages, can still search for intermediate solutions across different test functions, exhibiting significant continuous convergence capability and better final solution quality.

To verify the algorithm’s performance on ten-dimensional optimization problems, this paper conducted comparative experiments. The comparison results on CEC2022 test functions in ten dimensions are shown in Table 7. Through comparative analysis of performance on two-dimensional and ten-dimensional optimization problems, we found that the SAO algorithm exhibits significant advantages on ten-dimensional problems. Statistical data from Table 10 shows that SAO performs better than ElasticNet, Lasso, and Ridge on 8 functions each, showing inferiority to Lars on only 1 function. This indicates that as problem dimensionality increases, the SAO algorithm can better handle complex optimization problems and maintain high solution accuracy. In contrast, SAO’s performance on two-dimensional problems shows differentiated characteristics. For the three models, ElasticNet, Lasso, and Ridge, SAO performs better on 7 functions; however, it is better than Lars

Figure 3. Convergence plots of different algorithms.

Table 10. Comparison results of SAO and 4 models on CEC2022 test functions in 10 dimensions.

model on only 2 functions, forming a sharp contrast with the ten-dimensional case. It is particularly noteworthy that on function F3, SAO performs poorly against all compared models in the two-dimensional problem, while in the ten-dimensional problem, it shows only slight inferiority to some models. From the perspective of algorithm robustness, SAO’s variance is generally lower in ten-dimensional problems, indicating better stability. This dimensional adaptability suggests that the SAO algorithm possesses stronger search capability and convergence characteristics when handling high-dimensional optimization problems.