Figure 1 illustrates the multi-cell, multi-user, multi-server MEC system environment constructed in this study. The system consists of FNV controllers, end users, base stations and MEC servers deployed at the base stations. Each end-user in this MEC system periodically generates tasks. The FNV controller is the control centre of the cell MEC system and can monitor the resources of the MEC in that cell. MECs have more computing power than end users. The MEC server of a cell can make further requests to the FNV servers of other cells to be executed by the MEC servers of other cells and cannot be offloaded to the MEC.

1) Communication models

In the system model constructed in this research, the decision cycle is divided into a number of time slots t, where $t\in T$, $T=\left[1,2,\cdots ,T_{max}\right]$. All MEC servers in a cell can provide services to the users in the cell, but each end-user can only send tasks to one MEC server per time slot. The coordinates of the end users are defined as $L_{u_{i,j}}\left(t\right)=\left[x_{u_{i,j}},y_{u_{i,j}}\right]$ and the coordinates of the MEC servers are defined as $L_{ME{C}_{i,k}}=\left[x_{ME{C}_{i,k}},y_{ME{C}_{i,k}}\right]$. The system has s cells with b end users and m MEC servers in each cell. In this paper it is assumed that the location of the end users within the cell is random, but the MEC servers in the cell are fixed.

This study considers the use of Orthogonal Frequency Division Multiple Access (OFDMA) to implement the communication between users and MEC servers, and the MEC server links between different cells are connected by optical fibres. Next, assume that the kth MEC server is selected for the task. According to Shannon's formula, terminal $u_{i,j}$ transmits the task at a rate of

$Rat{e}_{i,j}\left(t\right)=W_{i,j}{\text{log}}_2\left(1+\frac{p_{i,j}^u{h}_{i,j}}{{\displaystyle \underset{c=1,c\ne j}{\overset{b}{\sum }}{p}_{i,c}^u{h}_{i,c}}+{\sigma }^2}\right)$ (1)

where W is the bandwidth of the subchannel, $p_{i,j}^u$ and $h_{i,j}^z$ are the received power and channel gain of the device in the channel, respectively. $\sigma$ is Gaussian noise.

The communication link in the system is assumed to be a line-of-sight link channel control, the channel gain can be expressed as:

$h_{i,j}^z=\frac{P_{i,j}^{up}}{\Vert L_{ME{C}_{i,k}}-L_{u_{i,j}}\left(t\right)\Vert }$ (2)

1) Computational modelling

A partial offloading model is considered in this study. The user offloads part of the computational tasks to the MEC server and further to other cells according to the offloading policy. Meanwhile, due to the use of optical fibre transmission between MEC servers, which has a high transmission rate that can reach 201.6 Gbps [15] . The system model in this study ignores the task backhaul delay and the inter-cell data transmission delay. It is assumed that the task selects the kth MEC server for offloading. The key components of the system experiment during offloading can be calculated as follows:

1) The time taken by the end user to transmit the task to the MEC server;

2) The time delay for the end user to process the task;

3) The time delay for the MEC in that cell to process the task;

4) The time delay for MECs in other cells to process the task.

The time at which the end user transmits the task to the MEC server can be expressed as

$T_{i,j}^{tra}\left(t\right)=\frac{\left(1-R_{i,j}^{rm}\left(t\right)\right)c_{i,j}\left(t\right)}{Rat{e}_{i,j}\left(t\right)}$ (3)

where $c_{i,j}\left(t\right)$ denotes the size of the task generated by the end user. $R_{i,j}^{rm}\left(t\right)$ denotes the end-user offload rate.

The end user processing delay is can be denoted as:

$T_{i,j}^L\left(t\right)=\frac{\left(1-R_{rm}\left(t\right)\right)c_{i,j}\left(t\right)z}{f_{i,j}^L}$ (4)

where z is the required calculation frequency per task unit. $f_{i,j}^L$ is the computation frequency of the user.

The delay of this cell MEC processing task can be expressed as

$T_{i,j,v}\left(t\right)=\frac{\left(1-R_{i,j}^{rs}\left(t\right)\right)R_{i,j}^{rm}\left(t\right)c_{i,j}\left(t\right)z}{{\theta }_{i,j}\left(t\right)f_{i,k}}$ (5)

where ${\theta }_{i,j}\left(t\right)$ denotes the proportion of computational resources of the MEC server to which the user is assigned, and $f_{i,k}$ the computational frequency of the MEC to which it is assigned. $R_{i,j}^{rs}\left(t\right)$ denotes the proportion of tasks transmitted to other cells.

The delay of the MEC server in the other cell to process the task can be expressed as:

$T_{i,j,ov}\left(t\right)=\frac{R_{rs}\left(t\right)R_{rm}\left(t\right)c_{i,j}\left(t\right)}{{{\theta }^{\prime }}_{i,l}\left(t\right)f_{i,k}}$ (6)

where ${{\theta }^{\prime }}_{i,l}\left(t\right)$ denotes the proportion of computing resources allocated to the task by MECs in other cells.

In this paper, it is assumed that the MEC server can only execute the task after the transmission is completed. At the same time, the transmission task starts when the end device starts executing the local task. At this time, the delay of each task is the maximum of the end-user computational delay, the computational delay of the MEC server and the transmission delay, which can be expressed as:

$T_{i,j}\left(t\right)=\max \left(T_{i,j}^L\left(t\right),T_{i,j}^{tra}\left(t\right)+T_{i,j,v}\left(t\right),T_{i,j,ov}\left(t\right)\right)$ (7)

2) Power model

The battery capacity of the MEC Server is indicated as $E_{i,k}^{battery}$. The MEC server consumes energy when serving end users and can be recharged with a charging efficiency of $E_{i,k}^{charge}$. Suppose the task selects the kth MEC server for offloading. The main components of the power consumption are as follows

1) Energy consumed by the end user

2) Energy consumed by the MEC server in the cell

3) Power consumed by other MECs in the cell

4) Power consumed by task transfer to the MEC server

5) Energy consumed when the MEC receives the task.

The energy consumption of the end user can be expressed as

$E_{i,j}^L\left(t\right)=P_{i,j}^L{T}_{i,j}^L\left(t\right)$ (8)

where $P_{i,j}^L$ denotes the power of the end user.

The energy consumption of the MEC server in this cell can be expressed as:

$E_{i,j,v}\left(t\right)=P_{i,k,v}{T}_{i,j,v}\left(t\right)$ (9)

where $P_{i,k,v}$ denotes the power of the selected MEC.

The energy consumption of the task transfer to the MEC server can be expressed as:

$E_{i,j}^{tra,up}\left(t\right)=P_{i,j}^{up}{T}_{i,j}^{tra}\left(t\right)$ (10)

where $P_{i,j}^{up}$ denotes the transmission power of the end user.

The energy consumption at the time of MEC receiving task can be expressed as:

$E_{i,j}^{tra,down}\left(t\right)=P_{i,k}^{down}{T}_{i,j}^{tra}\left(t\right)$ (11)

In terms of the MEC charging model, the residual of MEC at moment t can be expressed as:

$E_{i,k}^r\left(t\right)=\{\begin{array}{l}{E}_{i,j}^v\left(t\right),E_{i,j}^v\left(t\right)\le E_{i,k}^{battery}\\ E_{i,k}^{battery},E_{i,k}^{battery}>E_{i,j}^v\left(t\right)\end{array}$ (12)

Among others,

$E_{i,j}^v\left(t\right)=E_{i,k}^r\left(t-1\right)-E_{i,j}^{tra,down}\left(t-1\right)-E_{i,j,v}\left(t\right)+E_{i,k}^{charge}$ (13)

In summary, the total energy consumption of this system can be expressed as:

$E\left(t\right)={\displaystyle \underset{i=1}{\overset{s}{\sum }}{\displaystyle \underset{j=1}{\overset{b}{\sum }}{E}_{i,j}^{tra,down}\left(t\right)+E_{i,j}^{tra,up}\left(t\right)+E_{i,j,ov}\left(t\right)+E_{i,j,v}\left(t\right)+E_{i,j}^L\left(t\right)}}$ (14)

3) MEC storage model

In this paper, we consider a type of MEC server with limited memory resources. However, the offload rate of tasks transmitted from other users to other cells is 0. Therefore, the residual power of the MEC at time slot t can be expressed as follows:

$M_{i,k}^r\left(t\right)=\{\begin{array}{l}{M}_{i,k}^{memory},\\ M_{i,k}^r\left(t\right),M_{i,k}^r\left(t\right)\le M_{i,k}^{memory}\end{array}$ (15)

Among others,

$M_{i,k}^r\left(t\right)=M_{i,k}^r\left(t-1\right)-\left(1-R_{i,j}^{rs}\left(t\right)\right)R_{i,j}^{rm}\left(t\right)c_{i,j}\left(t\right)+{\displaystyle \underset{j=1}{\overset{b}{\sum }}{\theta }_{i,j}}{f}_{i,k}st$ (16)

In this paper, we study the optimisation objective of minimising the weighted sum of the system energy consumption and the average delay of the users. The joint optimisation problem can be expressed as:

$\underset{{\theta }_{i,j}\left(t\right),{{\theta }^{\prime }}_{i,j}\left(t\right),R_{i,j}^m\left(t\right),R_{i,j}^{rs}\left(t\right),R_{i,j}^{ism}\left(t\right),R_{i,j}^{iss}\left(t\right),t\in T}{\min }{\displaystyle \sum {\rho }_{1}E\left(t\right)+{\rho }_{2}T\left(t\right)}$ (17)

$\text{s}\text{.t}\text{.}\left[0,0\right]\le L_{u_{i,j}}\left(t\right)\le \left[L_{max,x},L_{max,y}\right]$ (18)

$0\le E_{i,k}^r\left(t\right)\le E_{i,k}^{battery}$ (19)

$1\le R_{ism}\left(t\right)\le m,R_{ism}\left(t\right)\in R,0\le R_{rm}\left(t\right)\le 1$ (20)

$1\le R_{iss}\left(t\right)\le s,R_{iss}\left(t\right)\in R,0\le R_{rs}\left(t\right)\le 1$ (21)

$0\le M_{i,k}^r\left(t\right)\le M_{i,k}^{memory}$ (22)

$0\le {\theta }_{i,j}\left(t\right)\le 1,{\displaystyle \underset{j=1}{\overset{b}{\sum }}{\theta }_{i,j}\left(t\right)}\le 1$ (23)

$0\le {{\theta }^{\prime }}_{i,j}\left(t\right)\le 1,{\displaystyle \underset{j=1}{\overset{b}{\sum }}{{\theta }^{\prime }}_{i,j}\left(t\right)}\le 1$ (24)

$T_{i,j}\left(t\right)\le T_{i,j,max}\left(t\right)$ (25)

where Equation (18) indicates that the location of the user in the cell cannot exceed the range of that cell, and Equation (19) indicates the energy limit of the MEC. Equation (20) indicates that the offloading rate is between 0 and 1 and the number of offloaded MECs cannot exceed the number of MEC servers in the cell. Equation (21) indicates that the offload rate is between 0 and 1 and the number of offloaded cells cannot exceed the total number of cells. Equation (22) indicates a range of memory resources for the MEC server. Equation (23) denotes a range in which the sum of the allocation ratios of the arithmetic resources and the ratios cannot exceed 1. Equation (24) represents a range where the sum of the allocation ratios of the arithmetic resources to the tasks passed from other cells cannot exceed 1 as well as the ratio. Equation (25) indicates that the delay of the task cannot exceed the maximum delay that the user can receive. After many experiments, ${\rho }_1=0.53$, ${\rho }_2=0.007$ can better guide the model to find the optimal policy.

The process of determining the optimal offloading strategy is modelled as an MDP problem in this study. The DRL algorithm is then employed to identify the optimal optimization strategy. The state space, action space and reward of the Markov decision process are presented in the subsequent section.

State space: the amount of tasks $c\left(t\right)$ and the location $L_u\left(t\right)$ of the user for each cell user in the cell, the remaining power $E^r\left(t\right)$ and the storage resource $M^r\left(t\right)$ of the MEC as the current state. Thus the state space can be represented as Equation (26):

$s\left(t\right)=\left\{c\left(t\right),L_u\left(t\right),E^r\left(t\right),M^r\left(t\right),M_{i,k}\left(t\right)\right\}$ (26)

Action space: this study considers offloading to a specific MEC server $R_{i,j}^{ism}\left(t\right)$, offloading rate $R_{i,j}^{rm}\left(t\right)$, whether to further offload to MEC servers $R_{i,j}^{ism}\left(t\right)$ and offloading rate $R_{i,j}^{rm}\left(t\right)$ in other cells as the actions of the intelligences. Therefore the action space can be expressed as Equation (27):

$a_{i,j}\left(t\right)=\left\{R_{i,j}^{ism}\left(t\right),R_{i,j}^{rm}\left(t\right),R_{i,j}^{iss}\left(t\right),R_{i,j}^{rs}\left(t\right)\right\}$ (27)

Reward: The optimization objective of this study is to minimize the weighted sum of the total delay $T\left(t\right)$ and the total energy consumption $E\left(t\right)$. The reward can be expressed as follows. Therefore the reward can be expressed as follows Equation (28).

$r\left(t\right)=-\left({\rho }_{1}E\left(t\right)+{\rho }_{2}T\left(t\right)\right)$ (28)

where, ${\rho }_1$ and ${\rho }_2$ denote the weights of energy consumption and delay respectively.

In this study, MEC server resources will be allocated after computational offloading. Effective allocation of resources can optimize the performance of the MEC system. This study considers the allocation of arithmetic resources.

The allocation of arithmetic resources refers to the allocation of arithmetic resources uploaded to the MEC server under the management of the NFV controller, with the main purpose of more fully meeting the needs of end users. The main purpose of the allocation of arithmetic resources is to reduce the processing delay of the user, and this study assumes that a user will only use one channel within a timestamp. Derivation of ${\theta }_{i,j}$ in $T_{i,j,v}$ shows that:

$\frac{\partial T_{i,j,v}}{\partial {\theta }_{i,j}}=-\frac{\left(1-R_{i,j}^{rs}\left(t\right)\right)R_{i,j}^{rm}\left(t\right)c_{i,j}\left(t\right)}{{\theta }_{i,j}{f}_{i,k}}$ (29)

It can be seen in the derivation of the above equation:

$\frac{{\partial }^2{T}_{i,j,v}}{\partial {\theta }_{i,j}^2}=2\frac{\left(1-R_{i,j}^{rs}\left(t\right)\right)R_{i,j}^{rm}\left(t\right)c_{i,j}\left(t\right)}{{\theta }_{i,j}^3{f}_{i,k}}\ge 0$ (30)

Since $R_{i,j}^{rs}\left(t\right)$, $R_{i,j}^{rm}\left(t\right)$, $f_{i,k}$ and ${\theta }_{i,j}$ are all greater than 0, it follows that the user can only pair the resources of one channel in a given timestamp. Consequently, the Hesse matrix of channel resources at that moment is positive definite. The optimal arithmetic resource allocation can be solved by Lagrange Multiplier Method as follows.

${\theta }_{i,j}^{*}=\frac{\sqrt{\left(1-R_{i,j}^{rs}\left(t\right)\right)R_{i,j}^{rm}\left(t\right)c_{i,j}\left(t\right)}}{{\displaystyle \underset{i=1}{\overset{m}{\sum }}\sqrt{\left(1-R_{i,j}^{rs}\left(t\right)\right)R_{i,j}^{rm}\left(t\right)c_{i,j}\left(t\right)}+\sqrt{s_{ov}}}}$ (31)

The optimal arithmetic allocation for tasks transmitted from other cells is thus determined:

${\theta }_{ov}^{*}=\frac{\sqrt{s_{ov}}}{{\displaystyle \underset{i=1}{\overset{m}{\sum }}\sqrt{\left(1-R_{i,j}^{rs}\left(t\right)\right)R_{i,j}^{rm}\left(t\right)c_{i,j}\left(t\right)}+\sqrt{s_{ov}}}}$ (32)

In this context, the symbol $s_{ov}$ represents the size of the task assigned to the kth MEC from other cells.

One of the main reasons why traditional reinforcement learning methods are difficult to apply to multi-intelligent body environments is that the strategy of each intelligent body is constantly changing during the training process, leading to an unstable environment for each individual intelligent body. MADDPG [21] is a reinforcement learning algorithm designed for multi-intelligent body systems. It extends DDPG for environments where multiple intelligences are present simultaneously. The learners of each intelligence share the experience playback buffer and use the strategies of other intelligences to assist in decision making, resulting in an overall collaborative behaviour. MADDPG uses a strategy of centralised training and decentralised execution in the training process.

In MADDPG, the actions of the intelligences affect the state transfer of the environment so that the state transfer function can be described as

$s_{t+1}=f\left(s_t,a_1,a_2,\cdots ,a_N\right)$ (33)

The reward function of MADDPG can be formulated as follows:

$r_t^i=R_i\left(s_t,a_1,a_2,\cdots ,a_N\right)$ (34)

where A denotes the reward given to, the ith intelligent in the tth timestamp, and B denotes the reward function of the intelligent.

The value function of MADDPG can be expressed as:

$Q_i\left(s_t,a_1,a_2,\cdots ,a_N\right)$ (35)

Algorithm 1 is a dynamic computational offloading algorithm based on MADDPG.

Algorithm 1. MADDPG-based dynamic computational offloading algorithm.

In this algorithm, the environment first interacts with each intelligence, and then the environment allocates tasks based on the actions returned by the intelligences. After all the intelligences have generated actions, the computational resources for different tasks are allocated based on the size of the tasks that are offloaded to the MEC server. Finally, the reward for each task is obtained.

In this study, the following schemes will be used as baseline algorithms for comparison tests:

1) Proximal Policy Optimization (PPO) algorithm

2) Deep deterministic policy gradient algorithm

3) Deep Q-network

The parameters required for the experimental simulation environment are shown in Table 1 and the parameters in the network are set in Table 2 .

This paper presents a series of simulation results to demonstrate the advantages of the MADDPGDCO algorithm proposed in the study. Firstly, the study explores the proportion of task offloading allocated to other cell MECs in the optimization strategy of the algorithm. Secondly, the simulation compares the MADDPGDCO algorithm proposed in this paper with the baseline algorithm used in the simulation in terms of delay, energy consumption and reward.

Firstly, the simulation studies the ratio of different task-based cell offloading and different cell offloading under this algorithm. As shown in Figure 1 , the algorithm proposed in this paper fully achieves the joint offloading of MEC servers in different cells. From Figure 2 , it can be seen that the MADDPGDCO algorithm proposed in this paper can effectively enable the MECs in different cells to process the tasks.

Secondly, the simulation examines the performance of delay, energy consumption and reward under different schemes. Figure 3 illustrates the delay of different schemes after convergence. It can be observed that the average delay is approximately 9.5 ms for MADDPGDCO, 14.5 ms for PPO, 11 ms for MADDPGDCO and 20.2 ms for DQN.

Figure 4 illustrates the energy consumption of different schemes following convergence. It can be observed that the average energy consumption of MADDPGDCO is approximately 600 units, PPO is approximately 900 units, DDPG is approximately 760 units and DQN is approximately 980 units.

Figure 5 illustrates the comparative reward outcomes between MADDPGDCO and the baseline algorithm. The reward plots for training with different scenarios demonstrate that MADDPGDCO outperforms the baseline algorithm.