Note that each player in AMDP takes its own actions based on the actions of other players. To capture these interdependence of actions of all players, we define the state vector at time $t$ as

$s_t={\left\{s_t^{n,m}\right\}}_{n=1,\cdots ,N;m=1,\cdots ,M}\cup \left\{{\tilde{a}}_t^{-n}\right\},$(4)

where $s_t^{n,m}$ is the $m$-th state of the $n$-th player at time $t$, $N$ is the number of players, and each player has $M$ states. Meanwhile, ${\tilde{a}}_t^{-n}$ denotes the most recent action taken by other players and is defined as

${\tilde{a}}_t^{-n}=\underset{N-1\text{actions}}{\underset{︸}{\left\{a_{t-N+1}^{n+1},\cdots ,a_{t-n}^N,a_{t-n+1}^1,\cdots ,a_{t-1}^{n-1}\right\}}},$(5)

where the superscript $-n$ means all players except Player $n$ and ${\tilde{a}}_t^{-n}$ has a total of $N-1$ actions. The specific composition of the state vector $s_t$ in Equation (4) is illustrated in Figure 2 for intuitive representation.

In addition, at the end of the strategic confrontation game we define the set of the game outcome as $c=\left\{c_1,c_2,\cdots ,c_W\right\}$, where $W$ is the numbers of the possible outcome. Each outcome $c_w\left(w\in \left\{1,\cdots ,W\right\}\right)$ of $c$ is uniquely associated with a distinct set of terminal states $Z_w\subset \mathcal{S}$, such that $Z_{w_1}\cap Z_{w_2}=\varnothing$ for any $w_1\ne w_2$ where $w_1$, $w_2\in \left\{1,2,\cdots ,W\right\}$. In other words, the game concludes with outcome $c_w$ if and only if the current state $s$ satisfies $s\in Z_w$, ensuring mutual exclusivity between outcomes.

Directly modeling high-dimensional continuous action spaces for multi-agent decision-making often incurs prohibitive computational costs and challenges in model convergence. To address this, we adopt a composite two-dimensional action space, which decomposes complicated actions into hierarchical decisions. Specifically, the action space is designed as a composite two-dimensional space

$a_t^n=\left(x_t^n,y_t^n\right),$(6)

where $x_t^n\in \left\{1,\cdots ,U\right\}$ and $y_t^n\in \left\{1,\cdots ,L\right\}$ represent the action type and action degree selected by Player $i$ at time $t$, $K$ and $L$ are the number of action types and action degrees, respectively. We notice that the action-space design in Equation (6) not only captures a diverse range of action types, but also quantifies their intensity, enabling the AMDP model to flexibly adapt to different levels of decision-making.

The state transfer function $p\left(s^{\prime }|s,a\right)$ is essentially a conditional probability of the state $s^{\prime }$ given $s$ and $a$, and is designed as the following normal distribution

$p\left(s^{\prime }|s,a\right)=\mathcal{N}\left(ℱ\left(s,a\right),{\sigma }^2\right),$(7)

where $ℱ\left(s,a\right)$ and ${\sigma }^2$ denote its mean and variance, respectively. To characterize the interdependencies between actions, the mean $ℱ\left(s,a\right)$ in Equation (7) is designed as

$ℱ\left(s,a\right)=ℱ\left(s_t^{n,m},{\tilde{a}}_t{}^{-n},a\right)=f_{\left(a_t^{-n},a\right)}\left(s_t^{n,m}\right),$(8)

where $f_{\left(a_t^{-n},a\right)}$ represents the state transition function determined by the $\left(a_t^{-n},a\right)$ tuple and can be set according to the actual physical meaning of the action in a specific scenario of strategic confrontation game.

In reinforcement learning, the reward function has an important role in guiding the sequential behavior of the agent, as it is usually regarded as the optimization criterion and ultimately shapes the learned policy for design-making. In this paper, the reward function for Player $n$, denoted as $r^n\left(s_t,a_t^n,s_{t+1}\right)$, consists of the following two components:

$r^n\left(s_t,a_t^n,s_{t+1}\right)=r_1^n\left(s_{t+1}\right)+r_2^n\left(s_t,s_{t+1}\right).$(9)

Here, the first term $r_1^n\left(s_{t+1}\right)$ is a terminal reward and defined as

$r_1^n\left(s_{t+1}\right)=R_w^n,\text{if}{s}_{t+1}\in Z_w,$(10)

where $R_w^n$ stands for the obtained reward value of Player $n$ when the state $s_{t+1}$ at time $t+1$ belongs to the $w$-th terminate state set $Z_w$. The second term $r_2^n\left(s_t,s_{t+1}\right)$ in Equation (9) reflects the state change-based reward, which encourages the progress of Player $n$ while penalizing the progress of other players and is defined as

$r_2^n\left(s_t,s_{t+1}\right)={\eta }_1{\displaystyle \underset{m=1}{\overset{M}{\sum }}\left(s_{t+1}^{n,m}-s_t^{n,m}\right)}-{\eta }_2{\displaystyle \underset{\begin{array}{c}j=1\\ j\ne n\end{array}}{\overset{N}{\sum }}{\displaystyle \underset{m=1}{\overset{M}{\sum }}\left(s_{t+1}^{j,m}-s_t^{j,m}\right)}},$(11)

where $N$ is the number of players, and $M$ is the number of state dimensions for each player, and ${\eta }_1$ and ${\eta }_2$ are the weighting factors of own-state gain and opponent-state loss. The reward design in Equation (9) encourages each player to maximize its own reward while suppressing the advancement of its opponents, thereby promoting competitive strategic behavior.

It is worth noting that the terminal reward $r_1$ in Equation (10) is usually assigned with a positive value to indicate a favorable outcome and a negative value to indicate an unfavorable one for the associated agent. This aligns with the long-term confrontation strategy of the associated agent with a hope of game success. The stepwise reward $r_2$ in Equation (11) provides additional guidance during the learning process by evaluating the quality of each state transition. In general, a positive reward is given if the post-transition state $s^{\prime }$ improves upon the previous transition state $s$. Such a reward design indeed encourages steady progress toward advantageous states while preserving the alignment with overarching strategic goals.

Based on the AMDP model introduced in Section 2, we implement a feed-forward neural network architecture parameterized by an appropriate set of parameters $\theta$, which adopts the state vector $s_t$ at time $t$ as input and obtains multiple estimated action-values $\left\{Q^{\prime }\left(s_t,a;\theta \right)\right\}$. It should be noted that when learning a neural network, the dimensions of input and output must satisfy the dimensions of the state vector and the number of actions, respectively. The relationship between the input $s_t$ and the output $Q^{\prime }\left(s_t,a;\theta \right)$ within deep neural network of the double Q network is detailed in Figure 4 . The parameter sets of $Q_1$ and $Q_2$ to be learned in the training phase are denoted by ${\theta }_1$ and ${\theta }_2$, respectively. Notice that the online network $Q_1$ and the target network $Q_2$ share the same neural network structure.

In reinforcement learning, unlike supervised learning, training data is not pre-collected but rather generated by agent through interactions with the environment and simultaneously is used to improve the agent’s policy through continuous learning. Specifically, at the beginning of training, the corresponding agent observes the current state $s$ and selects an action $a$ that maximizes the Q-value estimated by the online network $Q_1$ to interact with the environment though the state transfer function $p\left(s^{\prime }|s,a\right)$ and the reward function $r\left(s,a,s\right)$. This interaction yields the next state $s^{\prime }$ and a corresponding reward value $r$. Each interaction result is preserved in the experience replay buffer $D$ as a four-tuple sample $\left(s,a,r,s^{\prime }\right)$. Then, we take $N_b$ (batch size) samples used for training when the number of samples in the replay buffer reaches the maximum sample capacity $N_r$.

In interactions between the agent and the environment, we adopt an epsilon-greedy strategy to ensure that the agent can explore the action space sufficiently to avoid suboptimal solutions caused by premature convergence. Specifically, the epsilon-greedy strategy based action selection is given by

$a=\{\begin{array}{ll}\text{an}\text{action}\text{randomly}\text{selected}\text{from}\mathcal{A},\hfill & \text{with}\text{probability}ϵ,\hfill \\ \arg {\max }_a{Q}^{\prime }\left(s,a;{\theta }_1\right),\hfill & \text{with}\text{probability}1-ϵ,\hfill \end{array}$(12)

where $\mathcal{A}$ represents the action space, $s$ is the current state, and $Q^{\prime }\left(s,a;{\theta }_1\right)$ is the estimated action-value function that is used to predict the expected cumulative reward when taking action $a$ at state $s$. The value of $ϵ$ is decayed over time to allow for more exploration during the initial stages of training and gradually shift toward exploitation during the stable stage as the agent gains more confidence in its learned policy. Such a decay scheme in the $\lambda$-th episode is implemented as

$ϵ\left(\lambda \right)={ϵ}_{final}+\left({ϵ}_{start}-{ϵ}_{final}\right){\text{e}}^{-\frac{\lambda }{{ϵ}_{decay}}}$(13)

where ${ϵ}_{start}$ and ${ϵ}_{final}$ represent the value of $ϵ\left(\lambda \right)$ at the initial and the stable stage of training, respectively, and ${ϵ}_{decay}$ is used to control the decay rate of $ϵ\left(\lambda \right)$.

Based on the epsilon-greedy strategy in Equation (12), the corresponding agent performs actions in response to environmental states, thereby generating a sequence of four-tuple samples that serve as the training data. For each four-tuple sample, the online network $Q_1$ is employed to estimate the Q-values $Q^{\prime }\left(s^{\prime },a;{\theta }_1\right)$ for all possible actions with the next state $s^{\prime }$ and select an action that maximizes $Q^{\prime }\left(s^{\prime },a;{\theta }_1\right)$ with respect to $a$. The selected action in the online network $Q_1$ is then used in the target network $Q_2$ to compute the following the target Q-value

$Q^{\prime }\left(s^{\prime },\underset{a}{\arg \max }{Q}^{\prime }\left(s^{\prime },a;{\theta }_1\right);{\theta }_2\right).$(14)

Accordingly, we obtain the indirect estimate of the Q-value of the current state, which is given by the summation of the immediate reward $r$ and the discounted target Q-value of the next state, that is,

$y=r+\gamma Q^{\prime }\left(s^{\prime },\underset{a}{\arg \max }{Q}^{\prime }\left(s^{\prime },a;{\theta }_1\right);{\theta }_2\right),$(15)

where $\gamma \in \left(0,1\right]$ denotes the discount factor.

So far, we have obtained the direct estimate of Q-value $Q^{\prime }\left(s,a;{\theta }_1\right)$ and a more accurate indirect estimate $y$ of Q-value derived from the known reward $r$ and the direct estimate of the next step. Assume that a batch of $N_b$ samples are randomly drawn from the experience replay buffer $D$. Then, the loss function $ℒ$ of the neural network can be defined as the mean squared error between the two estimated Q values, that is,

$ℒ\left({\theta }_1\right)=\mathbb{E}\left[{\left(y-Q^{\prime }\left(s,a;{\theta }_1\right)\right)}^2\right].$(16)

The training (online network update) of the neural network parameter aims to find the optimal parameter ${\theta }_1^{*}$ by minimizing

${\theta }_1^{*}=\underset{{\theta }_1}{\arg \min }ℒ\left({\theta }_1\right).$(17)

It is worth noting that directly solving Problem (17) is generally intractable as the objective function $ℒ\left({\theta }_1\right)$ exhibits strong nonlinearity with respect to ${\theta }_1$. Therefore, we adopt a random gradient-based scheme to iteratively approximate the solution. Specifically, we propose to employ the Adam optimizer [34] , which has demonstrated strong performance for solving complicated nonlinear optimization inherent to deep reinforcement learning. To stabilize learning and suppress rapid fluctuations in the learning target, the parameter set ${\theta }_1$ of the online Q-network, $Q_1$, is periodically synchronized with that of the target Q-network, $Q_2$, i.e., ${\theta }_2^{*}={\theta }_1^{*}$, within each $N^{-}$ episodes.

Through sequentially interaction with the environment, the corresponding agent updates its policy to maximize the expected cumulative return. By decoupling the action selection from the target-value computation, the DDQN can effectively mitigate the Q-value overestimation bias, thereby enhancing training stability and boosting the ultimate policy decision-making performance. In conjunction with the AMDP formulation in Section 2, the entire DDQN based training procedure is outlined in Algorithm 1 , which is hereafter referred to as the AMDP-DDQN. The computational complexity of AMDP-DDQN in Algorithm 1 is dominated by $\mathcal{O}\left(N_{episode}{N}_{avg}{N}_b{N}_{state}\left({\displaystyle {\sum }_{i=1}^{N_H-1}{H}_i{H}_{i+1}}\right)N_{action}\right)$, where $N_{avg}$ denotes the number of averaging steps per episode, $N_{state}$ represents the state-space dimensionality, $N_{action}$ is the action-space cardinality, and $H_i$ corresponds to the $i$-th hidden layer dimensions in DDQN, $N_H$ indicates the number of hidden layers.

Algorithm 1. AMDP-DDQN.

Our experiments, detailed in the subsequent subsections, are conducted on a personal computer equipped with an NVIDIA GeForce RTX 3090 GPU, operating under Ubuntu 22.04. The specific implementation is carried out using Python 3.8, along with the PyTorch 1.12 deep learning framework and the Gym 2.6.0 reinforcement learning library.

The equal symmetry experiments are used to simulate a strategic confrontation game between two countries with equal strengths. As a result, the initial state settings of Country A and Country B are kept the same, as shown in Table 5 .

The reward evolution curves for Country A and Country B throughout the training process are presented in Figure 5 , where “Window size of 10 episodes” denotes the associated curve is obtained by a moving average with a window size of each 10 episodes. As illustrated, both agents (Countries) demonstrate a broadly similar trend: their rewards increase steadily from an initial value close to zero and gradually rise to approximately 25, after which both rewards plateau, indicating a convergence to a stable performance level. This consistent upward trajectory reflects the progressive improvement of agents in policy quality over time, suggesting that both have successfully acquired effective strategies through interaction with the environment. Notably, since the two countries share identical initial state configurations, it is reasonable to expect them to learn comparable optimal policies with a similar game result. In fact, we observe that the convergence of their reward curves further supports the theoretical expectation.

After completing the training phase, we carry out a series of experiments using the trained models of both agents to evaluate the game results of equal symmetry experiment. To evaluate the effectiveness of the learned strategies, four experimental scenarios are designed: 1) both countries adopt random strategies; 2) Country A adopts the model-based strategy while Country B uses a random strategy; 3) Country B adopts the model-based strategy while Country A uses a random strategy; and 4) both countries employ model-based strategies. The associated analysis of the game results is elaborated as follows.

Figure 6 illustrates the state evolution for Country A and Country B over multiple game rounds under different strategy combinations in a symmetric peer-to-peer experimental setting. The dark dashed lines represent the averaged value of each state across 1000 independent game simulations, while the shaded areas indicate the fluctuation ranges. A state value falling below zero indicates the termination of the game. As shown in Figure 6(a) , when both countries adopt random strategies, the evolution of their state values is nearly identical due to the fully symmetric initial conditions, demonstrating the fairness and symmetry of the experimental environment. In contrast, Figure 6(b) shows that when Country A adopts a model-based strategy while Country B keeps using a random strategy, the state values increase significantly, validating the effectiveness of the model-based strategy. A similar outcome is observed in Figure 6(c) , where Country B implements the model-based strategy and also experiences a notable improvement in its state values, further highlights the effectiveness and generalizability of the model-based approach. Finally, Figure 6(d) shows that when both countries adopt model-based strategies, a new equilibrium pattern emerges. This indicates that in intelligent strategic games, the mutual adoption of advanced strategies leads to a reestablishment of equilibrium, thereby reaffirming the inherently counterbalancing and adaptive nature of such interactions.

The game outcomes of equal symmetry experiment are depicted in Figure 7 . The blue bars represent the results when both countries employ random strategies. It is observed that wins and losses are distributed approximately evenly between the two countries, as expected under stochastic behavior. The orange bars correspond to the scenario where Country A utilizes a model-based strategy while Country B retains a random strategy. Compared to the fully random case, Country A experiences a marked reduction in failure rate, while Country B’s failures increase accordingly, indicating the strategic advantage gained from Country A. In contrast, the green bars illustrate the outcomes when Country B adopts a model-based strategy and Country A uses a random one. Here, Country B significantly reduces its failures, while Country A suffers more losses, reflecting the same pattern of strategy advantage. Finally, the red bars depict the case where both countries employ model-based strategies. Given their symmetric initial conditions, the game outcomes converge toward equilibrium, with each country failing approximately half of the time, further validating the competitive balance and mutual adaptation achieved through strategic learning.

The non-equal experiments are designed to simulate a strategic confrontation between two countries of unequal strengths. In this setting, Country A is assumed to possess a comprehensive advantage against Country B across all aspects in security, economy, technology, and administration. The corresponding initial state configurations for both countries are summarized in Table 6 . The reward evolution of each country during training, against the number of episodes, is illustrated in Figure 8 . Due to its dominant initial state, the reward curve of Country A quickly rises to approximately 30 and then stabilizes, reflecting rapid convergence to an effective strategy. In contrast, Country B, with disadvantages from its initial conditions, is only able to improve its performance gradually, with the reward curve rising from a negative value to approaching zero over time.

Figure 9 illustrates the evolution of state values for Country A and Country B under various strategy combinations, beginning from an asymmetric initial condition in which Country B starts with significantly lower state values than Country A. This experimental setting simulates a strategic confrontation scenario where a weaker nation competes against a stronger adversary. In Figure 9(a) , both countries adopt random strategies. Due to the considerable initial disadvantage, the state values of Country B fall into the failure region much earlier than that of Country A. This highlights that, under disadvantaged conditions, the weaker side is highly susceptible to rapid suppression if no effective strategy is employed. Figure 9(b) presents the case where Country A adopts a model-based strategy while Country B continues using a random strategy. Leveraging its initial advantage, the model of Country A learns to prioritize conservative and risk-averse actions that preserve its dominant state, resulting in consistently high state values and a stable trajectory toward success. This behavior reflects the model’s capacity to recognize and maintain strategic superiority through controlled decision-making.

In Figure 9(c) , Country B adopts a model-based strategy despite starting from a disadvantaged position. Although its initial state remains inferior, the learned policy enables B to take proactive steps to improve its condition and postpone failure. As a result, the state value of Country B shows significant improvement compared to cases of Figure 9(a) and Figure 9(b) . This indicates that even from a weak starting point, an effective strategy can prolong engagement and create potential opportunities. Figure 9(d) examines the scenario where both countries deploy model-based strategies. Despite the presence of strategic reasoning on both sides, the initial advantages of Country A enable its model to quickly identify and exploit this asymmetry by adopting an aggressive optimal policy. Consequently, Country B, although supported by a model, fails to attain an effective counter-strategy and suffers a swift decline in state values. This result illustrates a dominant amplification effect, where model-based strategies not only reinforce but intensify the impact of favorable initial conditions, allowing the dominant player to dictate the pace of the confrontation game and suppress any potential actions of the weaker side.

Overall, the results of Figure 9 demonstrate how initial conditions, when coupled with strategic learning, can significantly influence the dynamics of adversarial interactions. This may lead to irreversible trajectories shaped by early asymmetries.

The game results shown in Figure 10 reveal clear outcome disparities under different strategy combinations. When both countries adopt random strategies, Country A wins the majority of the games, owing to its initial advantage. This disparity becomes even more pronounced when Country A employs the model-based strategy, leading to an overwhelming dominance in which it secures nearly all victories. In contrast, when Country B adopts the model-based strategy while Country A uses a random strategy, the improvement in Country B’s performance is marginal. Despite the strategic upgrade, the significant disadvantage in its initial state limits the effectiveness of the model, resulting in only slightly better outcomes compared to using a random strategy. Finally, when both countries utilize model-based strategies, Country A still wins all the games, solely due to its strong initial advantage. This outcome underscores that, in highly asymmetric settings, strategic sophistication alone may be insufficient to overcome substantial disparities in starting conditions.

Finally, we validate the performance of the proposed AMDP-DDQN under the equal asymmetry condition, i.e., the two countries with roughly the same total strength but with different state values. Specifically, Country A is assigned stronger security power but a weaker economy, while Country B has the inverse situation: a stronger economy and weaker security power. The corresponding initial state configurations are detailed in Table 7 . The reward evolution versus the number of episodes during equal asymmetry experiment is depicted in Figure 11 , where the average reward in training of both countries is gradually rising and stable, in which country A is stable around 25 while country B is stable around 20. It is seen that under equal asymmetry conditions, both countries gradually learn effective strategies, with rewards stabilizing after initial fluctuations. Country A consistently achieves higher rewards than those of Country B, suggesting that the security advantage has a stronger impact on the outcomes. This indicates that even with equal total strength, the equal asymmetry condition significantly influences the final decision-making performance.

Figure 12 illustrates the state evolution of Country A and Country B under four strategy combinations in an equal-but-asymmetric setting, where we keep the same strategy combinations as in the previous experiments. In this experiment, both countries start with the same initial state value, but with different internal distributions of respective states: Country A begins with a relatively weaker economic state, while Country B is more vulnerable in its security domain.

In the scenario of Figure 12(a) , both countries adopt random strategies. Due to Country A’s initial economic weakness, its overall development capability remains limited, leading to a rapid decline in state value and a higher risk of failure. This indicates that even with equal total state values, structural weaknesses in some crucial initial state can enlarge the risk of defeat. In Figure 12(b) , Country A instead adopts a model-based strategy. The well-trained model of Country A first focuses on improving its economic state to ensure long-term sustainability, while also identifying and exploiting Country B’s security weakness. This dual strategy helps Country A delay its decline and obtain a temporary advantage, thereby demonstrating the adaptability and strategic precision of the training model in scenarios with asymmetric resource distribution.

In Figure 12(c) , Country B employs a model-based strategy. The training model of Country B accurately identifies the economic vulnerability of Country A, resulting in rapid defeat of Country A. This result shows that the training model can clearly recognize the main weakness of its opponent and respond with effective and targeted strategies. In Figure 12(d) , both countries adopt model-based strategies. In this experiment, both countries have their own weaknesses, but the economic weakness of Country A is more serious than the security weakness of Country B. As a result, when both countries use their trained model strategies to attack the weaknesses of the other one, Country A is more likely to be defeated first because its weakness is more critical.

In equal asymmetry experiment, both countries start with the same initial state value, but with different internal distributions of respective states. That is to say, Country A begins with a relatively weaker economic state, while Country B is more vulnerable in its security domain. We conclude that the equal-but-asymmetric setup reflects real-world situations where two agents with confrontation game have similar overall strength but differ in their resource distribution, which also leads to different strategic outcomes.

We now illustrate the game results of the equal asymmetry experimental games in Figure 13 . Note that when both agents (countries) adopt random strategies, while Country A has more economic failure than that of Country B and Country B has more security failure than that of Country A. This aligns with our initial state hypothesis that Country A has a weaker economic state and Country B has a weaker security state. In addition, when Country A uses the model strategy and Country B uses the random strategy, the game results show that the number of economic failure of Country A is greatly reduced while the number of security failure of Country B is greatly increased. This implies that Country A makes up for its economic weakness through actions while attacking the security weakness of Country B. Similarly, when Country B uses model strategy and Country A uses random strategy, the number of security failure of Country B decreases while the number of economic failures of Country A increases. This indicates that Country B also mitigates its own security weakness through effective actions of attacking the economic weakness of the other one. Finally, when both agents use the model strategy, we see that all of game outcomes exhibit failures due to their own weaknesses, and the number of failures of both countries are roughly equivalent.

To further investigate the model-based strategies which are learned from equal asymmetry training, the actions with the highest Q-values under different states are visualized using three-dimensional plots, as shown in Figure 14 and Figure 15 , where different colors represent different types of actions, and the color intensity indicates the action degree. In these figures, the x- and y-axes represent the current state levels of Country A and Country B, respectively, where each axis value denotes a uniform state setting across all four dimensions of the corresponding country. The z-axis indicates the opponent’s action in the previous round. It is observed from the two figures that Country A tends to choose defensive (green) actions most of the time, resorting to offensive or sanction actions only when the opponent is in a weakened state. In contrast, Country B often prefers sanction actions, leveraging its stronger economic position to gain an advantage.

Overall, all above experimental results clearly show that the AMDP-DDQN-trained models are able to make smart decisions and choose proper actions by considering both the current state and the previous action of the corresponding opponent. This allows each agent to adjust its strategy and respond effectively in different situations. Whatever facing symmetric or asymmetric scenarios, the associated models consistently choose actions that improve the chances of winning. Compared with random strategies, the model-based strategies usually lead to significantly superior outcomes, including higher rewards and longer survival times. This validates that the proposed AMDP-DDQN approach can successfully learn useful and adaptive policies, and that these well-learned strategies are effective across a variety of game settings.