The design of reward and punishment functions determines the good or bad degree of actions taken by mobile robot in a certain state [25]. Sparse rewards have always been a problem that affects the convergence of reinforcement learning

algorithms [26]. A continuous reward and punishment function is designed to solve the problem of sparse rewards. It is defined as follows:

r t = { r _ t a r r _ o b s D _ b e f o r e − D _ n o w μ ⋅ D _ n o w + d _ n o w − d _ b e f o r e η (8)

where:

r_tar represents the reward for reaching the target point;

r_obsrepresents the punishment for colliding an obstacle;

D_before represents the distance between the mobile robot and the target point before the action is performed;

D_now represents the distance between the mobile robot and the obstacle after performing the action;

d_now represents the shortest distance between the mobile robot and the obstacle at the current state;

d_before represents the shortest distance between the mobile robot and the obstacle before the action is performed;

μ and η represent normalized discount factors.

After the mobile robot performs the action, whether it is closer to the target point or farther from the obstacle, it will be rewarded according to the calculation Formula (8); if the robot is farther from the target point or closer to the obstacle, it will be punished.

Double BP Q-learning algorithm is proposed to improve the exploration-exploitation of ε-greedy strategy, and a dynamic ε-greedy strategy is designed. The ε-greedy strategy is used to balance the relationship between exploration and exploitation. Its essence is to explore and select actions with ε probability, and to learn the existed experience with 1-ε probability [27].

Given an initial value of exploitation ε₀ and peak value ε_end, with the increase of learning frequency, the exploration rate ε_t decreases from peak value ε_end to zero and remains unchanged, while the exploitation rate ε_r increases from the initial value ε₀ to peak value ε_endand remains unchanged. The exploration-exploitation rate is computed as follows:

ε r = { ε 0 + ε i ⋅ t _ s t e p , t _ s t e p < s t e p _ m a x ε _ e n d , t _ s t e p ≥ s t e p _ m a x ε t = { ε _ e n d − ε 0 − ε i ⋅ t _ s t e p , t _ s t e p < s t e p _ m a x 0 , t _ s t e p ≥ s t e p _ m a x (9)

where:

ε_irepresents the increase factor of exploration rate;

t_step represents the number of learning rounds;

step_max represents the maximum round of running.

The change of exploration-exploitation rate is shown in Figure 4.

Although random sampling breaks the correlation between samples, it ignores the importance of experience, and introduces the mechanism of priority experience replay to store the priority of samples through the binary tree structure Sumtree. High-priority samples are selected firstly to speed up learning [28].

Double BP Q-learning algorithm collects samples in the training process, which including state s_t, action a_t, reward r_t and state s_t+1 at the next moment. Samples (s_t, a_t, r_t, s_t+1) are stored in the experience pool. TD-error is used to define the sample priority, which is to estimate the difference between the Q values output by the estimation network and the target network. The larger the TD-error, the higher the corresponding priority [29]. The probability defined by the extracted sample is as follows:

P ( i ) = p i α ∑ i = 1 k p i α (10)

where:

αrepresents how much priority to add to the sample. If α is 0, it is an uniform sampling:

p i = 1 r a n k ( i ) (11)

where the rank(i) is sorted by TD-error.

The higher the sample priority, the higher the probability of the sample being extracted, and the lower the sample priority is guaranteed to have a certain probability of being selected.

Most tasks in local path planning by reinforcement learning are correlated, and parameters in related tasks are initialized by parameter migration in different map environments to speed up strategy learning of mobile robots in different scenarios [30].

Firstly, the pre-training model is loaded to obtain all the model parameters. The model parameters ω_s and ω_t to be trained are divided into two parts by using hierarchical Bayes. One part is the common task, such as the function ω_i, which tends to the target point. The other part is unique to each model parameters, such as dense discrete obstacles avoidance strategy v_s and special obstacles avoidance rule v_t. The transfer learning framework of this paper is as follows:

ω s = ω i + v s ω t = ω i + v t (12)

The model parameter ω_i approaching the target point is obtained through random initialization training, and it is initialized into the model parameter ω ′ i of the sparse discrete obstacles environment and the model parameter ω_t of the special obstacles environment. Then, combining with the special state operation rules set in Table 1 and the direction and distance information between the mobile robot and the target point, the obstacle avoidance strategy is learned. Finally, ω ′ i is initialized as the model parameter ω_s in the dense discrete obstacle environment to perfect the obstacles avoidance rules.

The structure diagram of transfer learning to local path planning is shown in Figure 5.

The mobile robot is trained by the designed Double BP Q-learning neural network and strategy designed above, and use the obtained model to plan local path.

The learning process of the algorithm is as follows.

Step 1. A standard blank map Q_Map is generated, and the number and position of starting points, target points and obstacles are randomly set. The initial direction is the connecting direction between the mobile robot and the target point.

Step 2. Initialize the parameters of Double BP Q-learning algorithm;

Step 3. The state s_t is obtained after preprocessing the map environment information returned by the sensors;

Step 4. Neural network parameters are initialized randomly in the initial training; otherwise, parameters are initialized by parameter transfer;

Step 5. The dynamic ε-greedy strategy is used to select the action a_t;

Step 6. The mobile robot performs the action a_t to obtain the reward r_t and the next state s_t+1;

Step 7. Store {s_t, a_t, r_t, s_t+1} in Memory bank as a replay unit;

Step 8. Extract small batch of experience for learning;

Step 9. If the next state is the termination state, the Q value of the target network is r_t; otherwise, the Q value is calculated through the target network as follows:

Q = r t + γ Q ( s t + 1 , arg max a Q ( s t + 1 , a s ; ω t ) ; ω t − ) (13)

Gradient descent algorithm is performed to calculate the Loss function as follows:

L o s s = ( r t + γ Q ( s t + 1 , arg max a Q ( s t + 1 , a s ; ω t ) ; ω t − ) − Q ( s t , a t ; ω t ) ) 2 (14)

Step 10. Update the estimation network parameter ω;

Step 11. Update the target network parameter ω- through copying ω to the ω- every C step.

The model parameters toward the target point are randomly initialized, and other parameters are initialized as shown in Table 2.

Using the set parameters, the mobile robot roamed in the environment without obstacles at the initial stage. After reaching the preset roaming times, small batches of experience are selected for learning. After the exploration rate rises to the preset peak value, the exploration rate remains unchanged. Continue training to the pre-training times and output the probability of successfully reaching the target point during parameters update process, the processes are shown in Figure 6.

Figure 7(a) and Figure 7(b) are the comparison of planned path toward the target point by Double Q-learning and Double BP Q-learning algorithms, respectively, where both the length of the paths is 20. The probability of the Double Q-learning algorithm successfully reaching the target point fluctuates greatly after the number of updating rounds p_step reaches 40 rounds and the model is unstable, as shown in Figure 6. However, the probability of Double BP Q-learning algorithm reaching the target point tends to be stable and reaches

more than 90% in the later period of parameter update, which shows the robot can reach that the target point, with a high probability in the obstacle-free map and the model generalization ability is higher.

The double BP Q-learning algorithm is used to train the model to complete the test of reaching the target point. Figure 8 shows two scenarios that the starting point and target point are arbitrarily set, and the optimal or relatively optimal path toward the target point can be planned through the trained model.

In the sparse discrete obstacles environment, model parameters approaching the target point are used as initialization to carry out model transfer. Some parameters are the same as parameters that approaching the target point, and other parameters are initialized as shown in Table 3.

By increasing the initial exploration rate ε₀ to 0.2, the mobile robot can make use of the learned rules earlier at the early stage of training and have a greater probability to reach the target point and reduce the unnecessary learning. Increase the number of initial roaming times and the memory bank capacity to make better use of the collected experience; after reaching the preset training

times, the probability of successfully reaching the target point in each round of parameter update in the sparse discrete obstacles environment is output. The training results of the two algorithms are shown in Figure 9.

Figure 10(a) and Figure 10(b) show the path comparison in the sparse discrete obstacles environment between the two algorithms, and the planned paths are all 21. As shown in Figure 9, when the number of parameter update p_step reaches 60 rounds, Compared with Double Q-learning algorithm, Double BP Q-learning algorithm has a greater probability of the robot reaching the target point, and the planned path is more smooth and has few redundant sections, which indicates that the Double BP Q-learning algorithm has learned part of the obstacles avoidance rules in the sparse discrete obstacles environment, and the output model has higher generalization ability.

After several tests, under different sparse discrete obstacles environments, the success rate of Double BP Q-Learning algorithm model for path planning can reach 76%, while the success rate of Double Q-Learning algorithm is only 65%. Figure 11 shows partial simulation results of the trained Double BP Q-learning algorithm in the sparse discrete obstacles environments.

In sparse discrete obstacles environment, due to the insufficient complexity of obstacle map, the model generalization ability is not high, and the strategy learned by the algorithm can’t deal with most of the maps, so the sparse discrete obstacles environment model is initialized to model parameters of the dense discrete obstacles environment through parameter transfer, some parameters are the same as parameters of the sparse discrete obstacles environment, other parameters are initialized as shown in Table 4.

Increase the number of roaming steps in the initial stage and reduce the increase extent of exploration rate. In the initial training process, the mobile robot can find more possible paths and learn more possible scenarios. At the same time, increase the memory bank capacity to store more different experiences. After reaching the preset peak value, the exploration rate remains unchanged and continues to train until reaching the preset training times and outputs the probability of successfully reaching the target point in each round of parameter update process. The simulation results of the two algorithms are shown in Figure 12.

However, as is shown in Figure 12, the probability of the Double Q-Learning algorithm reaching target point fluctuates greatly during the parameter update process, and the model is difficult to converge. However, the Double BP Q-learning algorithm can successfully reach the target point with a high probability after the number of parameter update rounds reaching 60 rounds, and the probability tends to be stable and the smoothness of the planned path is higher, which indicating that the model trained by the algorithm can be applied to most obstacles maps in dense discrete obstacles environments, and the generalization ability is stronger.

Figure 13(a) and Figure 13(b) are the comparison of the planned paths obtained by using the two algorithms respectively in a dense discrete obstacles environment, where the paths length planned by the algorithms are both 21.

After several tests, in the different dense discrete obstacles environments, the success rate of using Double BP Q-learning algorithm model to plan different maps can reach 89%, while the success rate of Double Q-learning algorithm is 82%. Figure 14 shows partial simulation results of the trained Double BP Q-learning model in the dense discrete obstacles environments.

In the special obstacles environment, there are some special scenarios that are easy to make the mobile robot enter the deadlock state. The model parameters approaching the target point are used as initialization to carry out parameter transfer. Some parameters are the same as those parameters approaching the target point, and other parameters are initialized as shown in Table 5.

Special obstacles environment mainly includes “U” type obstacles environment and “一” type obstacles environment. Mobile robot needs to avoid the special obstacles in advance or is able to escape after entering them. Through three sensors to detect obstacles information to set specific operation rules, as shown in Table 1, and combine the direction and distance information with the target point to make the mobile robot avoid entering deadlock area in advance. Reduce unnecessary study to speed up the convergence of the model. Figure 15 is the probability of reaching the target point after each round of parameter update successfully using the two algorithms.

Figure 16(a) and Figure 16(b) show the path comparison of special obstacles environment by the two algorithms, and the length is basically same. As is shown in Figure 15, the Double Q-learning algorithm converges faster in the early stage of training and has a higher probability of reaching the target point.

However, when the number of parameter update rounds reaches 50, the probability of reaching the target point is more than 90% for the model trained by the Double BP Q-learning algorithm. It tends to be stable and fluctuates less. It also has fewer redundant sections in the planned path, which indicates that the obstacles avoidance strategy learned can avoid certain deadlocks to reach the target point.

After several tests, the result is in line with the expected result of experiment. The simulation results of partial planned path under special obstacles environments using the trained Double BP Q-learning model are shown in Figure 17.