BLS is an efficient learning model developed by Chen et al. [21]. The model provides a powerful and effective learning framework, and its specific components and structure are shown in Figure 1. As seen in Figure 1, the structure of BLS is straightforward and similar to the random vector functional-link neural network (RVFLNN) [25]. Compared to deep learning models, which has only one hidden layer and few model parameters.

The creation of a standard BLS can be succinctly described as follows.

1) Let the training dataset be { ( X , Y ) | X ∈ ℝ N × M , Y ∈ ℝ N × C } . Since each feature mapping in the BLS model contains F neurons and there are n feature mappings ϕ i ( ⋅ ) from the input data X , the relevant formula is:

where W e i ∈ ℝ M × F and β e i ∈ ℝ N × F are randomly generated. However, the W e i in 1) is typically adjusted by sparse autoencoder in order to eliminate the consequences of randomness. It is equivalent to solving the following problems.

Figure 1. Structure of broad learning system (BLS).

where λ is a constant parameter. Based on the above solution, we can get the sparse autoencoder solution W e i , and then obtain all the feature nodes as Z n = [ Z 1 , ⋯ , Z n ] in the feature layer.

2) After that, Z n goes through a nonlinear transformation to generate enhancement nodes. The following formula can be used to define the output matrix of the k th enhancement node:

where W h k ∈ ℝ n F × E and β h k ∈ ℝ N × E are also randomly generated, ζ j ( ⋅ ) is a nonlinear function. The presentation of all the enhancement nodes is H m = [ H 1 , ⋯ , H m ] .

3) Finally, we obtain the output weights of the BLS model by concurrently combing Z n and H m . Thus, the final form of the BLS model is as follows:

where A = [ Z n | H m ] = [ Z 1 , ⋯ , Z n | H 1 , ⋯ , H m ] ; W stands for the output weight matrix, which can be calculated by W = A + Y .

In general, we also think about integrating an elastic-net regularizer with BLS to obtain a more robust model. This model is termed elastic-net BLS (ENBLS), and its specific problems are as follows:

Consider J network agents, which have the ability to communicate with their neighbors and perform some local computations. One can think of the communication network as a directed graph G ( J , ℰ ) , where J = { 1 , ⋯ , J } serving as the corresponding set of vertices and ℰ represents the connections between agents. N j in means the collection of in-neighbors of agent j , N j out stands for the group of out-neighbors of agent j .

Based on this strategy, this paper focuses on resolving a parameter learning problem for ENBLS on the above multi-agent network. In particular, the agents work together to tackle the following optimization problem:

Our goal is that each agent j in the network cooperatively seeks the optimal weight matrix W for problem (6) when the underlying graph G is directed.

The AB/Push-Pull algorithm [19][20] is a technique that can tackle distributed optimization problems on directed graphs . It is often utilized to solve distributed optimization problems as follows:

where x is the global decision variable, each objective f j : ℝ p → ℝ is convex.

Assumption 3.1Each f j is μ -strongly convex and its gradient is L-Lipschitz continuous. i.e., for any x , x ′ ∈ ℝ p

As a result, the only optimal solution x * for the problem (7) can be found under the condition that Assumption 3.1 is satisfied. By recalling the AB/Push-Pull algorithm in [19] and [20], we know that in this algorithm, each node j maintains two variables: x j ( k ) and D j ( k ) . The role of x j ( k ) is used to estimate the optimal value, while D j ( k ) is employed in monitoring the average gradient. The specific iterative procedure of this algorithm is as follows:

where ρ j ≥ 0 stands for step size, R and C satisfy the conditions of the following assumption.

Assumption 3.2The matrix R ∈ ℝ J × J is nonnegative row stochastic, and C ∈ ℝ J × J is nonnegative column stochastic, i.e., R 1 = 1 , 1 ⊤ C = 1 ⊤ .

It is worth noting that the procedure of Directed-DENBLS includes decentralized learning for feature fine-tuning and decentralized parameter learning of the output weights.

Thus, Directed-DENBLS first designs decentralized feature fine-tuning over directed graphs. We consider a directed graph of J agents interconnected. Next, the matrices Z i and W e i are replicated and taken over by { Z i , j , W e i , j } j = 1 J . In this way, we introduce consensus constraints W e i , j = W e i , j ′ and break problem (2) into the following group of sub-problems:

Recalling the advantages of the AB/Push-Pull technique, we use this algorithm for the mentioned optimization problem. We first construct R and C , where R and C satisfy Assumption 3.2. Next, an auxiliary local variable, D j , is introduced so that the following procedure can solve the optimization problem (10).

where ∇ f j ( W e i , j ) = Z i , j ⊤ ( Z i , j W e i , j − X j ) + λ J sign ( W e i , j ) . Based on the result of (11)-

(12), we find that at the kth iteration, agent j computes ∑ j ′ = 1 J R j j ′ ( W e i , j ′ ( k ) ) , which aims to average the information received from its neighbors. The following step is to add the gradient difference term ∇ f j ( W e i , j ( k + 1 ) ) − ∇ f j ( W e i , j ( k ) ) to D j ( k + 1 ) , together with the initialization condition eventually shows that it tracks the average gradient across the network [26]. Therefore, the expected results of decentralized feature fine-tuning can be obtained immediately by (11)-(12). Then the nodes of the feature layer can be recomputed with this outcome as Z j n = [ Z ′ 1 , j , ⋯ , Z ′ n , j ] = [ X j W e 1 , j ⊤ , ⋯ , X j W e n , j ⊤ ] . As soon as that is done, Z j n undergoes a nonlinear transformation by (3) to produce the enhancement nodes H j m = [ H ′ 1 , j , ⋯ , H ′ m , j ] for the enhancement layer. In the end, we put the Z j n and H j m together to create A j , i.e., A j = [ Z j n | H j m ] = [ Z ′ 1 , j , ⋯ , Z ′ n , j | H ′ 1 , j , ⋯ , H ′ m , j ] .

In what follows, we focus on solving the output weights of the model output layer. To begin with, let us reformulate the problem in (6) by introducing a collection of local variables W j , j = 1 , ⋯ , J . Thereby, problem (6) is reformulated as a distributed problem.

(13)

As in the first phase of the process, in the second step of Directed-DENBLS, we also introduce the AB/Push-Pull algorithm to address the mentioned optimization problem. Similarly, the extra variable B j plays a role in monitoring the gradient. In the same way, the variables for each iteration k are iteratively updated by the following procedure based on the AB/push-pull algorithm:

where ∇ f j ( W j ) = A j ⊤ ( A j W j − Y j ) + λ α J sign ( W j ) + 2 λ ( 1 − α ) J W j . R and C satisfy the conditions of the Assumption 3.2.

Likewise, from the (14), we know that at each iteration k , agent j receives local decision variables W j ′ from its neighbors. Moreover, according to (15), agent j transmits B j to its neighbors. Thus, from (14) and (15), we derive the expected performance of Directed-DENBLS. Finally, to illustrate the Directed-DENBLS algorithm more clearly, the process of Directed-DENBLS is described in detail in Algorithm 1.

In this section, a few unique acceleration algorithms are introduced. We start by reformulating the optimization problem (16) as

where f j ( W j ) = ∑ j = 1 J ( 1 2 ‖ A j W j − Y j ‖ 2 2 + λ α J ‖ W j ‖ 1 + λ ( 1 − α ) J ‖ W j ‖ 2 2 ) . ABm: Distributed heavy-ball (ABm) [27] extends AB by introducing Polyaks heavy-ball

momentum to accelerate the convergence of AB and Push-Pull. The iterations of the ABm algorithm are as follows:

where β ≥ 0 stands for momentum parameter.

ABN: ABN [28] is an accelerated version of another AB algorithm. This algorithm adds Nesterov’s momentum method which seeks to improve the convergence and applies it to arbitrary graphs. The specific procedure of the ABN algorithm is as follows:

where β k is momentum parameter.

Cleveland: Cleveland is a heart disease dataset. The dataset is divided into two major categories, namely: with and without heart disease.

Quake: Quake describe the relationship between quake and focal_depth integer, latitude real, longitude real, richter real.

WBC: The Breast Cancer Wisconsin (WBC) dataset are divided into two classes: patients with benign symptoms or malignant diagnoses.

Optdigits: The Optdigits is a 10 classification dataset with a total of 5620 samples. The image data are represented by an 8 × 8 pixel matrix with 64 pixel dimensions.

In the experiments of this paper, we consider a connected directed graph consisting of 4 agents, and the weight matrices R and C are built on the same basis graph, where G R and G C are shown in Figure 2. R and C are created by the following corresponding equations.

Figure 2. Communication network.

where d m a x i n is the maximum in-degree, and d m a x o u t is the maximum out-degree. And L R and L C are the Laplace matrices corresponding to G R and G C .

The step size ρ for all experiments is 10⁻⁵ and the regularization parameter λ is 2⁻³⁰. The experimental results are the average of fifty trials to reduce random errors.

In the experiments with the Quake dataset, the results of the three algorithms with the same network structure for 500 iterations are shown in Figure 3(a) and Table 2. From the experimental results in Figure 3(a), we can see that ABN and ABm converge faster than AB. AB converges after 100 iterations, while ABN and ABm converge after about 50 iterations. The numerical results in Table 2 show that the REMSE of the three algorithms are the same.

(a) Quake

(b) Cleveland

Figure 3. The results of RMSE for the AB/Push-Pull, ABN and ABm algorithms.

Table 2. RMSE for different algorithms on the test datasets.

For Cleveland, we considered a 20-20-300 model structure for all algorithms. The results of 500 iterations for the three algorithms are shown in Figure 3(b) and Table 2. The Table 2 shows that the RMSE of AB is smaller than that of ABN and ABm. And Figure 3(b) demonstrates that AB, ABN, and ABm converge after about 50 iterations, but it is obvious that ABN and ABM onverge faster than AB.

For WBC dataset, the simulation outcomes are displayed in Figure 4(a) and Table 3. From the results in Table 3, we can see that the MCRs of the three algorithms are similar, but from the simulation results in Figure 4(a), ABN converges after 200 iterations and converges faster than AB. Figure 5 compares the test confusion matrices of ABN and ABm. ABN has an overall accuracy of 91.3%, while ABm achieves 91.6% with fewer misclassified samples.

Table 3. MCR of different algorithms on the test datasets (%).

(a) WBC

(b) Optdigits

Figure 4. The results of MCR for the AB/Push-Pull, ABN and ABm algorithms.

(a) ABN

(b) ABm

Figure 5. The results of confusion matrix for the ABN and ABm algorithms (WBC).

For Optdigits, a 10 classified dataset, we took a 30-5-500 model structure and compared the MCRs of the three algorithms. From the results in Table 3, we can see that AB and ABN have the same MCR and ABm has the smallest MCR. From the simulation outcomes in Figure 4(b), after 800 iterations, ABm converges faster than the other two algorithms, while ABN and AB have the same trend. Figure 6 compares the test confusion matrices of ABN and ABm on the Optdigits dataset. ABN achieves an overall accuracy of 90.1%, while ABm reaches 91.1% with fewer misclassified samples.

(a) ABN

(b) ABm

Figure 6. The results of confusion matrix for the ABN and ABm algorithms (Optdigits).