Consider an episodic Markov decision process (MDP) $M=\left(\mathcal{S},\mathcal{A},\mathbb{P},T,R,\gamma \right)$, where $\mathcal{S}$ is the state space, $\mathcal{A}$ is the action space, $\mathbb{P}\left(s^{\prime }|s,a\right)$ is the transition distribution, $T$ is the length of the episode, $R\left(s,a\right)$ is the reward function, and $\gamma$ is the discount factor. For a stochastic policy $\pi \left(a|s\right):\mathcal{S}\times \mathcal{A}\to ℝ$, action-value function $Q^{\pi }$ is defined as

$Q^{\pi }\left(s,a\right):={\mathbb{E}}_{\pi }\left[\underset{t=0}{\overset{\infty }{{\displaystyle \sum }}}{\gamma }^{t}R\left(s_t,a_t\right)\right]|a_t~\pi \left(\cdot |s_t\right),s_{t+1}~\mathbb{P}\left(\cdot |s_t,a_t\right),s_0=s,a_0=a.$

Standard RL aims at learning the optimal policy ${\pi }^{\star }$ such that $Q^{{\pi }^{\star }}\left(s,a\right)\ge Q^{\pi }\left(s,a\right)$ for all $s\in \mathcal{S},a\in \mathcal{A}$ and all $\pi$. The corresponding $Q$-function of the policy satisfies the Bellman operator:

${\mathcal{T}}^{\pi }Q\left(s,a\right)=\mathbb{E}\left[R\left(s,a\right)\right]+\gamma \cdot {\mathbb{E}}_{\mathbb{P},\pi }\left[Q\left(s^{\prime },a^{\prime }\right)\right].$

In the offline setting, the agent is not allowed to interact with the environment [18] . The objective of agents is to learn an optimal policy only from a fixed dataset $\mathcal{D}:=\left\{\left(s,a,r,s^{\prime }\right)\right\}$. For all states $s\in \mathcal{D}$, let $\hat{\pi }\left(a|s\right):=\frac{{{\displaystyle \sum }}_{s^{\prime },a^{\prime }\in \mathcal{D}}{1}_{\left(s^{\prime }=s,a^{\prime }=a\right)}}{{{\displaystyle \sum }}_{s^{\prime }\in \mathcal{D}}{1}_{s^{\prime }=s}}$ be the empirical behavior policy. In order to avoid the situation where the denominator of the fraction is 0 in the theoretical analysis, we assume that $\hat{\pi }\left(a|s\right)>0$. Broadly, actions that are not drawn from $\hat{\pi }$ (i.e., those with low probability density) are the out-of-distribution (OOD).

Distributional RL directly models the full distribution of returns $Z^{\pi }\left(s,a\right)={\displaystyle {\sum }_{t=0}^{\infty }{\gamma }^{t}R\left(s_t,a_t\right)}$, instead of merely learning its expected value [19] . The distributional Bellman operator for policy evaluation is ${\mathcal{T}}^{\pi }Z\left(s,a\right):\stackrel{D}{=}R\left(s,a\right)+\gamma Z\left(s^{\prime },a^{\prime }\right)$, where $\stackrel{D}{=}$ indicates equality in distribution. Define the quantile function be the inverse of the cumulative density function $F_Z\left(z\right)=Pr\left(Z<z\right)$ as $F_Z^{-1}\left(\tau \right):=\text{inf}\left\{z\in ℝ:\tau \le F_Z\left(z\right)\right\}$ [20] , where $\tau$ denotes the quantile fraction. For random variables $U$ and $V$ with quantile functions $F_U^{-1}$ and $F_V^{-1}$, the $p$-Wasserstein distance $W_p\left(U,V\right)={\left({\displaystyle {\int }_0^1{\left|F_U^{-1}\left(\tau \right)-F_V^{-1}\left(\tau \right)\right|}^p\text{d}\tau }\right)}^{1/p}$ is the $L_p$ metric on quantile functions. [19] gave that the distributional Bellman operator ${\mathcal{T}}^{\pi }$ is a $\gamma$-contraction in the $W_p$, i.e., let ${\bar{d}}_p\left(Z_1,Z_2\right):={\sup }_{s,a}{W}_p\left(Z_1\left(s,a\right),Z_2\left(s,a\right)\right)$ be the largest Wasserstein

distance over $\left(s,a\right)$, and be the space of distributions over $ℝ$ with bounded $p$-th moment, then

${\bar{d}}_p\left({\mathcal{T}}^{\pi }{Z}_1,{\mathcal{T}}^{\pi }{Z}_2\right)\le \gamma \cdot {\bar{d}}_p\left(Z_1,Z_2\right)\forall Z_1,Z_2\in \mathcal{Z}.$

Fitted distributional evaluation (FDE) [15] approximates ${\mathcal{T}}^{\pi }$ by ${\hat{\mathcal{T}}}^{\pi }$ using $\mathcal{D}$, then $Z^{\pi }$ can be estimated by starting from an arbitrary ${\hat{Z}}^0$ and iteratively computing

$\begin{array}{l}{\hat{Z}}^{k+1}=\underset{Z}{\arg \min }{ℒ}_p\left(Z,{\hat{\mathcal{T}}}^{\pi }{\hat{Z}}^k\right),\\ \text{where}{ℒ}_p\left(Z,Z^{\prime }\right)={\mathbb{E}}_{\mathcal{D}\left(s,a\right)}\left[W_p{\left(Z\left(s,a\right),Z^{\prime }\left(s,a\right)\right)}^p\right].\end{array}$

In distributional RL, let risk measure function $\Phi :\mathcal{Z}\to ℝ$ be a map from the value distribution space to real numbers. Given a distorted function $g\left(\tau \right)$ over $\left[0,1\right]$, the distorted expectation of $Z$ is

${\Phi }_g\left(Z\left(s,a\right)\right)={\displaystyle {\int }_0^1{F}_{Z\left(s,a\right)}^{-1}}\left(\tau \right)g\left(\tau \right)\text{d}\tau ,$

and the corresponding policy is ${\pi }_g\left(s\right):=\arg {\max }_a{\Phi }_g\left(Z\left(s,a\right)\right)$ [13] . Specially, if $g=\text{Uniform}\left(\left[0,1\right]\right)$, then $Q^{\pi }\left(s,a\right)={\Phi }_g\left(Z\left(s,a\right)\right)$. For other choices of $g\left(\tau \right)$, please refer to Section 5.2.

Robust RL learns the policy by introducing worst-case adversarial noise to the system dynamics and formulating the noise distribution as the solution of a zero-sum minimax game. In order to learn robust policy $\pi$, SR²L [21] obtains $\hat{s}=\arg {\max }_{\hat{s}\in {\mathbb{B}}_d\left(s,ϵ\right)}{D}_{\text{J}}\left(\pi \left(\cdot |s\right)\left|\right|\pi \left(\cdot |\hat{s}\right)\right)$ by adding a perturbation to the state $s$, where ${\mathbb{B}}_d\left(s,ϵ\right)=\left\{\hat{s}:d\left(s,\hat{s}\right)\le ϵ\right\}$ and the Jeffrey’s divergence $D_{\text{J}}$ for two distributions P and $Q$ is defined by $D_{\text{J}}\left(P\left|\right|Q\right)=\frac{1}{2}\left[D_{\text{KL}}\left(P\left|\right|Q\right)+D_{\text{KL}}\left(Q\left|\right|P\right)\right]$, and then define a smoothness regularizer for policy as ${ℛ}_s^{\pi }={\mathbb{E}}_{s\sim {\rho }^{\pi }}{\max }_{\hat{s}\in {\mathbb{B}}_d\left(s,ϵ\right)}{D}_{\text{J}}\left(\pi \left(\cdot |s\right)\left|\right|\pi \left(\cdot |\hat{s}\right)\right)$. Analogously, ${ℛ}_s^Q={\mathbb{E}}_{s~{\rho }^{\pi }}{\max }_{\hat{s}\in {\mathbb{B}}_d\left(s,ϵ\right)}{\left(Q\left(s,a\right)-Q\left(\hat{s},a\right)\right)}^2$ is the smoothness regularizer for Q-function, where ${\rho }^{\pi }$ is state distribution.

In this part, we sample three sets of state-action pairs and form three different loss functions to obtain a conservative smooth policy. First of all, we construct a perturbation set ${\mathbb{B}}_d\left(s,ϵ\right)$ to obtain $\left(\hat{s},a\right)$ pairs, where ${\mathbb{B}}_d\left(s,ϵ\right)$ is an $ϵ$-radius ball measured in metric $d\left(\cdot ,\cdot \right)$ and $\hat{s}\in {\mathbb{B}}_d\left(s,ϵ\right)$. Then we sample $\left(s,\hat{a}\right)$ pairs from the current policy ${\pi }_{\varphi }$, where $\hat{a}~{\pi }_{\varphi }\left(s\right)$. ORDER contains $M$ Z-function and denotes the parameters of the $m$-th Z-function and the target Z-function as ${\theta }_m$ and ${{\theta }^{\prime }}_m$, respectively. With the help of the constructions, we will give the different learning targets for $\left(s,a\right)$, $\left(\hat{s},a\right)$ and $\left(s,\hat{a}\right)$ pairs, respectively.

SAC [7] incorporates an entropy term in its objective function to optimize both cumulative rewards and policy stochasticity, enhancing policy robustness and generalization. We also introduce an entropy term, for a $\left(s,a\right)$ pair sampled from $\mathcal{D}$, we obtain the target as

${\hat{\mathcal{T}}}_{\text{DS}}^{\pi }{Z}_{{\theta }_m}\left(s,a\right):=R\left(s,a\right)+\gamma \left[\underset{j=1,\cdots ,M}{\text{min}}{Z}_{{{\theta }^{\prime }}_j}\left(s^{\prime },a^{\prime }\right)-c\cdot \log \pi \left(a^{\prime }|s^{\prime }\right)\right],$

where the next Z-function takes the minimum value among the target Z-functions. The loss function is defined as follows,

${ℒ}_Z\left({\theta }_m\right)={ℒ}_p\left(Z_{{\theta }_m},{\hat{\mathcal{T}}}_{\text{DS}}^{\pi }{Z}_{{\theta }_m}\right).$

Next, we introduce the smoothness regularizer term to the distribution action-value function that is designed to enhance the smoothness of the Z-function. Specifically, we minimize the difference between $Z_{{\theta }_m}\left(\hat{s},a\right)$ and $Z_{{\theta }_m}\left(s,a\right)$, where $\left(\hat{s},a\right)$ is a state-action pair with a perturbed state. Then we take an adversarial $\hat{s}\in {\mathbb{B}}_d\left(s,ϵ\right)$ which maximizes ${ℒ}_p\left(Z_{{\theta }_m}\left(\hat{s},a\right),Z_{{\theta }_m}\left(s,a\right)\right)$. The final smooth term we introduced is shown below:

${ℒ}_{\text{smooth}}\left(s,a;{\theta }_m\right)=\underset{\hat{s}\in {\mathbb{B}}_d\left(s,ϵ\right)}{\max }\varrho \cdot {ℒ}_p\left(Z_{{\theta }_m}\left(\hat{s},a\right),Z_{{\theta }_m}\left(s,a\right)\right),$(1)

where $\varrho \in \left[0,1\right]$ is a factor that balances the learning of in-distribution and out-of-distribution values. Thus, for the selected $\hat{s}\in {\mathbb{B}}_d\left(s,ϵ\right)$, we minimize Equation (1) to get a smooth Z-function. Since the actions should be near the offline data and close to the behavior actions in the dataset, we do not consider OOD action for smoothing in this part.

Finally, we consider the following loss function to prevent overestimation of OOD actions.

${ℒ}_{\text{OOD}}\left({\theta }_m,\beta \right)=\beta \cdot {\mathbb{E}}_{U\left(\tau \right),\mathcal{D}\left(s,a\right)}\left[c_0\left(s,a\right)\cdot F_{Z_{{\theta }_m}\left(s,a\right)}^{-1}\left(\tau \right)\right],$

for some state-action dependent scale factor $c_0$ and $U=\text{Uniform}\left(\left[0,1\right]\right)$.

Incorporating both the in-distribution target and OOD target, we conclude the loss function in ORDER as follows,

$\underset{{\theta }_m}{\min }{\mathbb{E}}_{s,a,r,s^{\prime }~\mathcal{D}}\left[{ℒ}_Z\left({\theta }_m\right)+\alpha {ℒ}_{\text{smooth}}\left(s,a;{\theta }_m\right)+{ℒ}_{\text{OOD}}\left({\theta }_m,\beta \right)\right].$(2)

With the above smooth limits, we can learn a robust policy with fewer policy changes under perturbations. We choose a state $\hat{s}\in {\mathbb{B}}_d\left(s,ϵ\right)$ as mentioned above which is maximizing $D_{\text{J}}\left({\pi }_{\varphi }\left(\cdot |s\right)\left|\right|{\pi }_{\varphi }\left(\cdot |\hat{s}\right)\right)$. Consequently, our loss function for policy is as follows:

$\underset{\varphi }{\text{min}}\left[{\mathbb{E}}_{s,a,r,s^{\prime }~\mathcal{D}}\left[-\underset{j=1,\cdots ,M}{\text{min}}{\Phi }_g^j\left(s,a\right)+{\alpha }_1\underset{\hat{s}\in {\mathbb{B}}_d\left(s,ϵ\right)}{\text{max}}{D}_{\text{J}}\left({\pi }_{\varphi }\left(\cdot |s\right)\left|\right|{\pi }_{\varphi }\left(\cdot |\hat{s}\right)\right)+\log {\pi }_{\varphi }\left(a|s\right)\right]\right],$ (3)

where the first term is designed to get a conservative policy by maximizing the minimum of the distributional functions ensemble and the last term is a regularization term.

In this subsection, we integrate the distributional evaluation and policy improvement algorithms introduced in Section 3.1 and Section 3.2 into the actor-critic framework. With the loss function introduced in Equation (2), it is natural to get the following iterative formula for $Z^{\pi }$ by starting from an arbitrary ${\tilde{Z}}^0$,

$\begin{array}{c}{\tilde{Z}}^{k+1}=\underset{Z}{\arg \min }\{{ℒ}_p\left(Z\left(s,a\right),{\hat{\mathcal{T}}}_{\text{DS}}^{\pi }{\hat{Z}}^k\left(s,a\right)\right)+\underset{\hat{s}}{\text{max}}\varrho \cdot {ℒ}_p\left(Z\left(\hat{s},a\right),Z\left(s,a\right)\right)\\ +\beta \cdot {\mathbb{E}}_{U\left(\tau \right),\mathcal{D}\left(s,a\right)}\left[c_0\left(s,a\right)\cdot F_{Z\left(s,a\right)}^{-1}\left(\tau \right)\right]\}.\end{array}$(4)

Following [17] , we suggest employing a min-max objective in which the inner loop selects the current policy to maximize the objective, while the outer loop is responsible for minimizing the objective with respect to this policy:

$\begin{array}{c}{\tilde{Z}}^{k+1}=\underset{Z}{\arg \min }\underset{\mu }{\text{max}}\{\beta \cdot {\mathbb{E}}_{U\left(\tau \right)}\left[{\mathbb{E}}_{\mathcal{D}\left(s\right),\mu \left(\tilde{a}|s\right)}{F}_{Z\left(s,\tilde{a}\right)}^{-1}\left(\tau \right)-{\mathbb{E}}_{\mathcal{D}\left(s,a\right)}{F}_{Z\left(s,a\right)}^{-1}\right]\\ +{ℒ}_p\left(Z\left(s,a\right),{\hat{\mathcal{T}}}_{\text{DS}}^{{\pi }^k}{\hat{Z}}^k\left(s,a\right)\right)+\underset{\hat{s}}{\text{max}}\varrho \cdot {ℒ}_p\left(Z\left(\hat{s},a\right),Z\left(s,a\right)\right)\},\end{array}$

where $\mu$ is an actor policy. To establish a well-posed optimization problem, we introduce a regularization term in the original objective. Detailed analysis procedures are provided in Appendix A.1. The final optimization objective becomes

$\begin{array}{c}{\tilde{Z}}^{k+1}=\underset{Z}{\arg \min }\{\beta \cdot {\mathbb{E}}_{U\left(\tau \right)}\left[{\mathbb{E}}_{\mathcal{D}\left(s\right)}\log \underset{a}{{\displaystyle \sum }}\exp \left(F_{Z\left(s,a\right)}^{-1}\left(\tau \right)\right)-{\mathbb{E}}_{\mathcal{D}\left(s,a\right)}{F}_{Z\left(s,a\right)}^{-1}\left(\tau \right)\right]\\ \stackrel{}{}+{ℒ}_p\left(Z\left(s,a\right),{\hat{\mathcal{T}}}_{\text{DS}}^{{\pi }^k}{\hat{Z}}^k\left(s,a\right)\right)+\underset{\hat{s}}{\text{max}}\varrho \cdot {ℒ}_p\left(Z\left(\hat{s},a\right),Z\left(s,a\right)\right)\},\end{array}$

where $U=\text{Uniform}\left(\left[0,1\right]\right)$. To perform optimization with respect to the distribution $Z$, we express the quantile function using a neural network $G_{\theta }\left(\tau ;s,a\right)\approx F_{Z\left(s,a\right)}^{-1}\left(\tau \right)$. In order to calculate ${ℒ}_p\left(Z,Z^{\prime }\right)=W_p{\left(Z,Z^{\prime }\right)}^p$ [22] , we minimize the weighted pairwise Huber regression loss of various quantile fractions. $\tau$-Huber quantile regression loss [23] with threshold $\kappa$ is represented as

${ℒ}_{\kappa }\left(\delta ;\tau \right)=\{\begin{array}{ll}\left|\tau -1\left\{\delta <0\right\}\right|\cdot {\delta }^2/\left(2\kappa \right),\hfill & \text{if}\left|\delta \right|\le \kappa ,\hfill \\ \left|\tau -1\left\{\delta <0\right\}\right|\cdot \left(\left|\delta \right|-\kappa /2\right),\hfill & \text{otherwise},\hfill \end{array}$

where $\tau ,{\tau }^{\prime }~U$ are random quantiles, $\delta =r+\gamma G_{{\theta }^{\prime }}\left({\tau }^{\prime };s^{\prime },a^{\prime }\right)-G_{\theta }\left(\tau ;s,a\right)$, $\left(s,a,r,s^{\prime }\right)~\mathcal{D}$, $a^{\prime }~\pi \left(\cdot |s^{\prime }\right)$. More details for the algorithm ORDER are presented in Algorithm 1.

Before presenting our theorems, similar to [15] , we first give some assumptions about the MDP and dataset. Next, we assume that the search space in Equation (4) includes all possible functions.

Assumption 1. For all $s\in \mathcal{S}$ and $a\in \mathcal{A}$, $F_{Z^{\pi }\left(s,a\right)}$ is smooth. Furthermore, the search space of the minimum over Z in Equation (4) is overall smooth functions $F_{Z\left(s,a\right)}$.

The assumption is given to guarantee the boundness of the $p$-th moments of $Z^{\pi }\left(s,a\right)$ as well as $Z^{\pi }\in \mathcal{Z}$.

Assumption 2. For all $s\in \mathcal{S}$ and $a\in \mathcal{A}$, there exists $\zeta \in {ℝ}_{>0}$, such that $F_{Z^{\pi }\left(s,a\right)}$ is $\zeta$-strongly monotone, i.e., ${F^{\prime }}_{Z^{\pi }\left(s,a\right)}\left(x\right)\ge \zeta$.

This assumption is only designed to ensure the convergence of $F_{Z^{\pi }\left(s,a\right)}^{-1}\left(x\right)$ in our theoretical analysis. In algorithm implementation, we introduce a policy smoothing term and choose the minimum value of the Z-function in the ensemble to restrict the policy update amplitude, thus implicitly satisfying monotonicity. Next, we assume that the infinite norm of two quantiles with perturbation is restricted by a fixed constant.

Assumption 3. For all $s\in \mathcal{S}$, $a\in \mathcal{A}$ and the selected $\hat{s}\in {\mathbb{B}}_d\left(s,ϵ\right)$, we assume that ${\Vert F_{Z\left(\hat{s},a\right)}^{-1}-F_{Z\left(s,a\right)}^{-1}\Vert }_{\infty }\le \sigma$, $\sigma$ is constant.

In the above assumption, ${\Vert \cdot \Vert }_{\infty }$ represents the infinite norm. Since perturbed observation $\hat{s}$ is randomly sampling from an ${\ell }_{\infty }$ ball of norm $ϵ$, the assumption is reasonable.

Finally, we assume $p>1$. Therefore, we derive the following lemma which characterizes the conservative distribution soft evaluation iterates $Z\left(s,a\right)$ with the distributional soft Bellman operator.

Lemma 1. Suppose Assumptions 1 - 3 hold. For all $s\in \mathcal{D}$, $a\in \mathcal{A}$, $k\in \mathbb{N}$, and $\tau \in \left[0,1\right]$, we have $F_{Z\left(s,a\right)}^{-1}\left(\tau \right)=F_{{\hat{\mathcal{T}}}_{\text{DS}}^{\pi }{\hat{Z}}^k\left(s,a\right)}^{-1}\left(\tau \right)-c\left(s,a\right)$, where $c\left(s,a\right)={\left|\beta p^{-1}{c}_0\left(s,a\right)\pm {\sigma }^{p-1}\right|}^{\frac{1}{p-1}}\cdot \text{sign}\left(c_0\left(s,a\right)\right)$.

For detailed proof, please refer to Appendix B.2. Briefly, it is according to the result of a simple variational skill to handle that F is a function, and setting the derivative of Equation (4) equal to zero.

Next, we define the conservative soft distributional evaluation operator ${\tilde{\mathcal{T}}}^{\pi }={\mathcal{O}}_c{\hat{\mathcal{T}}}_{\text{DS}}^{\pi }$ by compositing ${\hat{\mathcal{T}}}_{\text{DS}}^{\pi }$ and the shift operator ${\mathcal{O}}_c:\mathcal{Z}\to \mathcal{Z}$, which is defined by $F_{{\mathcal{O}}_{c}Z\left(s,a\right)}^{-1}\left(\tau \right)=F_{Z\left(s,a\right)}^{-1}\left(\tau \right)-c\left(s,a\right)$. Consequently, it is following that ${\tilde{Z}}^{k+1}={\tilde{\mathcal{T}}}^{\pi }{\tilde{Z}}^k$.

Now, the main theorem we came up with shows that the conservative distributional soft evaluation obtains a conservative quantile estimate of the true quantile at all quantiles $\tau$.

Theorem 1. For any $\delta \in {ℝ}_{>0},c_0\left(s,a\right)>0$, with probability at least $1-\delta$, we have

$F_{Z^{\pi }\left(s,a\right)}^{-1}\left(\tau \right)\ge F_{{\tilde{Z}}^{\pi }\left(s,a\right)}^{-1}\left(\tau \right)+{\left(1-\gamma \right)}^{-1}\underset{s^{\prime },a^{\prime }}{\text{min}}\left\{c\left(s^{\prime },a^{\prime }\right)-\frac{1}{\zeta }\sqrt{\frac{5\left|\mathcal{S}\right|}{n\left(s^{\prime },a^{\prime }\right)}\text{log}\frac{4\left|\mathcal{S}\right|\left|\mathcal{A}\right|}{\delta }}\right\},$

for all $s\in \mathcal{D},a\in \mathcal{A}$, and $\tau \in \left[0,1\right]$. Furthermore, for $\beta \ge {\text{max}}_{s,a}\left\{\frac{p\left(\text{Δ}{\left(s,a\right)}^{p-1}+{\sigma }^{p-1}\right)}{c_0\left(s,a\right)}\right\}$, we have $F_{Z^{\pi }\left(s,a\right)}^{-1}\left(\tau \right)\ge F_{{\tilde{Z}}^{\pi }\left(s,a\right)}^{-1}\left(\tau \right)$.

For detailed proof, please refer to Appendix B.2. As the theorem shows, the above inequality indicates that the quantile estimates obtained by ${\tilde{\mathcal{T}}}^{\pi }$ are a lower bound of the true quantiles. Furthermore, we give a sufficient condition to show that the result given in Theorem 1 is not a vacuous conclusion. Therefore, Theorem 1 theoretically illustrates that ORDER does not exacerbate the distribution shift problem, and the mitigation of the distribution shift problem will be demonstrated in the experimental section.

The performance of many RL algorithms will exhibit different behaviors under different distorted expectations. Consequently, we can acquire the same conservative estimates of these objectives, which is a kind of generalization of Theorem 1.

Corollary 1 For any $\delta \in {ℝ}_{>0}$, $c_0\left(s,a\right)>0$, sufficiently large $\beta$ and $g\left(\tau \right)$, with probability at least $1-\delta$, for all $s\in \mathcal{D},a\in \mathcal{A}$, we have ${\Phi }_g\left(Z^{\pi }\left(s,a\right)\right)\ge {\Phi }_g\left({\tilde{Z}}^{\pi }\left(s,a\right)\right)$.

In particular, $Q^{\pi }\left(s,a\right)\ge {\tilde{Q}}^{\pi }\left(s,a\right)$ is obtained if we take $g=\text{Uniform}\left(\left[0,1\right]\right)$. By choosing different risk measure functions, we can apply this conclusion to any risk-sensitive offline RL.

The cause of the distribution shift problem is that the distribution of collected offline training data differs from the distribution of data in practice. BCQ [9] addresses this problem through policy regularization techniques, which formulates the policy as an adaptable deviation constrained by maximum values [32] . One solution to alleviate the distribution shift problem as mentioned in BEAR [33] is incorporating a weighted behavior-cloning loss achieved by minimizing maximum mean discrepancy (MMD) into the policy improvement step. Though learning a conservative Q-function caused by distribution shift, CQL [17] solves the overestimation of value functions, which theoretically proves that a lower bound of the true value is obtained. With the introduction of distributional reinforcement learning into offline RL, CODAC [15] learns a conservative return distribution by punishing the predicted quantiles returned for the OOD actions. From another perspective, continuous quantiles are used in MQN-CQR [16] to learn the quantile of return distribution with non-crossing guarantees. ORDER builds on these approaches, but considers the entropy regularizer of quantile function instead of an unchangeable constant for ensuring sufficient exploration and may relieve the training imbalance.

Owning to the distribution shift issues, current offline RL algorithms tend to prioritize caution in their approach to value estimation and action selection. Nevertheless, this selection can compromise the robustness of learned policies, making them highly sensitive to even minor perturbations in observations. As a groundbreaking piece of work, SR²L [21] achieves a more robust policy by introducing a smoothness regularizer into both the value function and the policy. A robust Q-learning algorithm proposed in REM [30] is presented that integrates multiple Q-value networks in a random convex combination of multiple Q-value estimates, ensuring that the final Q-value estimate remains robust. With arbitrarily large state spaces, RFQI [34] learns the optimal policy by employing function approximation using only an offline dataset and addresses robust offline reinforcement learning problems. In ORDER, we import the smoothing regularizer to the distribution functions and policies instead of simple action-value functions.

Our method is related to the previous offline RL algorithms based on constraining the learned value function [17] [35] . What sets our method apart is that it can better capture uncertain information about OOD actions and learn more robust policies with the introduction of the smoothing technique into distributional reinforcement learning. In addition, we enhance the network by using the entropy regularizer of quantile networks and increasing the number of network ensembles, which also improves robustness.

In all the aforementioned datasets, we compare our method against several previous popular offline RL algorithms, including 1) bootstrapping error accumulation reduction (BEAR) [33] , 2) conservative q-learning (CQL) [17] , 3) CODAC [15] , 4) robust offline reinforcement learning (RORL) [36] , 5) monotonic quantile network with conservative quantile regression (MQN-CQR) [16] . The results of BEAR and CQL are directly taken from [18] . For CODAC and MQN-CQR, their results are taken from the original paper. Since the RORL paper does not report scores with five random seeds, we run the RORL using the official code base. Implementation details are given in Appendix D. Table A1 and Table A2 list the hyperparameters of ORDER in different datasets. Without loss of generality, we employ the neutral risk measure in this subsection.

The performance of all these algorithms is exhibited in Table 1 , which reports the average normalized scores along with their corresponding standard deviations. We observe that ORDER outperforms BEAR in all tasks and surpasses the performance of the CQL algorithm. Significantly, our algorithm surpasses the performance of current distributed offline RL methods (see CODAC and MQN-CQR in Table 1 ), which is attributed to the assurance of robustness in ORDER. Meanwhile, ORDER competes favorably with the current state-of-the-art algorithms, owing to the safety guaranteed by distributional RL.

In this subsection, we investigate how the risk measure functions affect the performance of ORDER. We compare three risk-averse learned policies [14] in distribution RL with the risk-neutral measure function. Specifically, for different $g\left(\tau \right)$, three ways of distorted expectation are considered,

Besides, we evaluate three risk-seeking learned policies.

The results are shown in Table 2 and Table 3 , indicating that there is little difference between risk-averse methods and risk-seeking learned policies. This suggests that risk measure functions within the ORDER framework are not highly sensitive. At this point, we also empirically demonstrate the robustness of our approach.

Without loss of generality, we choose the hopper-medium-replay-v2 dataset as an example to conduct the ablation study in this subsection. The performance of our ORDER algorithm under different Ms is visualized in Figure 2 . We observe with the increase of M, the effect has a significant improvement in both computation efficiency and stability; as shown in the yellow and purple lines. However, M should not be too large, which is presumably attributed to the overfitting problem (see the blue line, the normalized score value fluctuates significantly around the training epoch of 700). In conclusion, M is set as four to balance between robustness enhancement and computational efficiency improvement.

In this section, we provide a comprehensive account of our practical implementation of ORDER, offering a detailed explanation of the process.

To establish a well-defined optimization problem, we introduce a regularization term denoted as $ℛ\left(\mu \right)$ in the original objective:

$\begin{array}{c}{\hat{Z}}^{k+1}=\underset{Z}{\arg \min }\underset{\mu }{\text{max}}\{\beta \cdot {\mathbb{E}}_{U\left(\tau \right)}\left[{\mathbb{E}}_{\mathcal{D}\left(s\right),\mu \left(\tilde{a}|s\right)}{F}_{Z\left(s,\tilde{a}\right)}^{-1}\left(\tau \right)-{\mathbb{E}}_{\mathcal{D}\left(s,a\right)}{F}_{Z\left(s,a\right)}^{-1}\left(\tau \right)\right]\\ +{ℒ}_p\left(Z,{\hat{\mathcal{T}}}^{{\pi }^k}{\hat{Z}}^k\right)+\underset{\hat{s}}{\text{max}}\varrho \cdot {ℒ}_p\left(Z\left(\hat{s},a\right),Z\left(s,a\right)\right)\}+ℛ\left(\mu \right).\end{array}$

Let $c_0\left(s,a\right)=\frac{\mu \left(a|s\right)-\hat{\pi }\left(a|s\right)}{\hat{\pi }\left(a|s\right)}$ and $ℛ\left(\mu \right)$ to be the entropy $\mathscr{H}\left(\mu \right)$, then $\mu \left(a|s\right)\propto \text{exp}\left(Q\left(s,a\right)\right)=\text{exp}\left({\displaystyle {\int }_0^1{F}_{Z\left(s,a\right)}^{-1}}\left(\tau \right)\text{d}\tau \right)$ is the solution to the inner-maximization. Substituting this selection into the previously mentioned regularized objective function gives

$\begin{array}{c}{\hat{Z}}^{k+1}=\underset{Z}{\arg \min }\{\beta \cdot {\mathbb{E}}_{U\left(\tau \right)}\left[{\mathbb{E}}_{\mathcal{D}\left(s\right)}\log \underset{a}{{\displaystyle \sum }}\exp \left(F_{Z\left(s,a\right)}^{-1}\left(\tau \right)\right)-{\mathbb{E}}_{\mathcal{D}\left(s,a\right)}{F}_{Z\left(s,a\right)}^{-1}\left(\tau \right)\right]\\ \stackrel{}{}+{ℒ}_p\left(Z,{\hat{\mathcal{T}}}^{{\pi }^k}{\hat{Z}}^k\right)+\underset{\hat{s}}{\text{max}}\varrho \cdot {ℒ}_p\left(Z\left(\hat{s},a\right),Z\left(s,a\right)\right)\}.\end{array}$

As demonstrated in [15] , we also introduce a parameter $\zeta \in {ℝ}_{>0}$ to threshold the quantile value difference between $\mu$ and $\hat{\pi }$, and give this difference a weight $\xi \in {ℝ}_{>0}$. Then we get a trainable expression of $\beta$ through the process of dual gradient descent:

$\begin{array}{l}\underset{Z}{\min }\underset{\beta \ge 0}{\max }\{\beta \cdot {\mathbb{E}}_{U\left(\tau \right)}\left[\xi \cdot \left[{\mathbb{E}}_{\mathcal{D}\left(s\right)}\log \underset{a}{{\displaystyle \sum }}\exp \left(F_{Z\left(s,a\right)}^{-1}\left(\tau \right)\right)-{\mathbb{E}}_{\mathcal{D}\left(s,a\right)}{F}_{Z\left(s,a\right)}^{-1}\left(\tau \right)\right]-\zeta \right]\\ \begin{array}{c}\\ \end{array}+{ℒ}_p\left(Z,{\hat{\mathcal{T}}}^{{\pi }^k}{\hat{Z}}^k\right)+\underset{\hat{s}}{\text{max}}\varrho \cdot {ℒ}_p\left(Z\left(\hat{s},a\right),Z\left(s,a\right)\right)\}.\end{array}$

Since all our experiments take place in continuous-control domains, it is not feasible to list all possible actions as and directly compute $\log \underset{a}{{\displaystyle \sum }}\exp \left(F_{Z\left(s,a\right)}^{-1}\left(\tau \right)\right)$. In our implementation, we employ the importance sampling approximation method described in [17] , and obtain

$\begin{array}{l}\log \underset{a}{{\displaystyle \sum }}\exp \left(F_{Z\left(s,a\right)}^{-1}\left(\tau \right)\right)\\ \approx \log \left(\frac{1}{2C}\underset{a_i\sim U\left(\mathcal{A}\right)}{\overset{N}{{\displaystyle \sum }}}\left[\frac{\exp \left(F_{Z\left(s,a\right)}^{-1}\left(\tau \right)\right)}{U\left(\mathcal{A}\right)}\right]+\frac{1}{2C}\underset{a_i\sim \pi \left(a|s\right)}{\overset{N}{{\displaystyle \sum }}}\left[\frac{\exp \left(F_{Z\left(s,a\right)}^{-1}\left(\tau \right)\right)}{\pi \left(a_i|s\right)}\right]\right),\end{array}$(5)

where $U\left(\mathcal{A}\right)=\text{Uniform}\left(\mathcal{A}\right)$ represents a uniform distribution of actions. Algorithm 7.1 summarizes a single step of the actor and critic updates used by ORDER.

Proof. By the definition of $p$-Wasserstein distance, we can re-write Equation (4) as

$\begin{array}{l}{\mathbb{E}}_{\mathcal{D}\left(s,a\right)}{\displaystyle {\int }_0^1{\left|F_{Z\left(s,a\right)}^{-1}\left(\tau \right)-F_{{\hat{\mathcal{T}}}_{\text{DS}}^{\pi }{\hat{Z}}^k\left(s,a\right)}^{-1}\left(\tau \right)\right|}^p\text{d}\tau }+\beta \cdot {\mathbb{E}}_{U\left(\tau \right),\mathcal{D}\left(s,a\right)}\left[c_0\left(s,a\right)\cdot F_{Z\left(s,a\right)}^{-1}\left(\tau \right)\right]\\ +{\mathbb{E}}_{\mathcal{D}\left(s,a\right),\hat{s}\in {\mathbb{B}}_d\left(s,ϵ\right)}{\displaystyle {\int }_0^1{\left|F_{Z\left(\hat{s},a\right)}^{-1}\left(\tau \right)-F_{Z\left(s,a\right)}^{-1}\left(\tau \right)\right|}^p\text{d}\tau }\\ ={\displaystyle {\int }_0^1{\mathbb{E}}_{\mathcal{D}\left(s,a\right)}}[{\left|F_{Z\left(s,a\right)}^{-1}\left(\tau \right)-F_{{\hat{\mathcal{T}}}_{\text{DS}}^{\pi }{\hat{Z}}^k\left(s,a\right)}^{-1}\left(\tau \right)\right|}^p+\beta \cdot c_0\left(s,a\right)\cdot F_{Z\left(s,a\right)}^{-1}\left(\tau \right)\\ +{\left|F_{Z\left(\hat{s},a\right)}^{-1}\left(\tau \right)-F_{Z\left(s,a\right)}^{-1}\left(\tau \right)\right|}^p]\text{d}\tau .\end{array}$

For arbitrary smooth functions ${\varphi }_{s,a}$ with compact support $\left[V_{\text{min}},V_{\text{max}}\right]$, we consider a perturbation $G_{s,a}^{\epsilon }\left(\tau \right)=F_{Z\left(s,a\right)}^{-1}\left(\tau \right)+\epsilon \cdot {\varphi }_{s,a}\left(\tau \right)$ of $F_{Z\left(s,a\right)}^{-1}\left(\tau \right)$, then the above formula can be written as

${\displaystyle {\int }_0^1{\mathbb{E}}_{\mathcal{D}\left(s,a\right)}}\left[{\left|G_{s,a}^{\epsilon }\left(\tau \right)-F_{{\hat{\mathcal{T}}}_{\text{DS}}^{\pi }{\hat{Z}}^k\left(s,a\right)}^{-1}\left(\tau \right)\right|}^p+\beta \cdot c_0\left(s,a\right)\cdot G_{s,a}^{\epsilon }\left(\tau \right)+{\left|G_{\hat{s},a}^{\epsilon }\left(\tau \right)-G_{s,a}^{\epsilon }\left(\tau \right)\right|}^p\right]\text{d}\tau .$

Then the following equation is obtained considering the derivative of $\epsilon$ at $\epsilon =0$:

$\begin{array}{l}{\frac{\text{d}}{\text{d}\epsilon }{\displaystyle {\int }_0^1{\mathbb{E}}_{\mathcal{D}\left(s,a\right)}}\left[{\left|G_{s,a}^{\epsilon }\left(\tau \right)-F_{{\hat{\mathcal{T}}}_{\text{DS}}^{\pi }{\hat{Z}}^k\left(s,a\right)}^{-1}\left(\tau \right)\right|}^p+\beta c_0\left(s,a\right)\cdot G_{s,a}^{\epsilon }\left(\tau \right)+{\left|G_{\hat{s},a}^{\epsilon }\left(\tau \right)-G_{s,a}^{\epsilon }\left(\tau \right)\right|}^p\right]\text{d}\tau |}_{\epsilon =0}\\ ={\mathbb{E}}_{\mathcal{D}\left(s,a\right)}{\displaystyle {\int }_0^1[p{\left|F_{Z\left(s,a\right)}^{-1}\left(\tau \right)-F_{{\hat{\mathcal{T}}}_{\text{DS}}^{\pi }{\hat{Z}}^k\left(s,a\right)}^{-1}\left(\tau \right)\right|}^{p-1}\text{sign}\left(F_{Z\left(s,a\right)}^{-1}\left(\tau \right)-F_{{\hat{\mathcal{T}}}_{\text{DS}}^{\pi }{\hat{Z}}^k\left(s,a\right)}^{-1}\left(\tau \right)\right)}\\ +\beta c_0\left(s,a\right)+p{\left|F_{Z\left(\hat{s},a\right)}^{-1}\left(\tau \right)-F_{Z\left(s,a\right)}^{-1}\left(\tau \right)\right|}^{p-1}\text{sign}\left(F_{Z\left(\hat{s},a\right)}^{-1}\left(\tau \right)-F_{Z\left(s,a\right)}^{-1}\left(\tau \right)\right)]{\varphi }_{s,a}\left(\tau \right)\text{d}\tau .\end{array}$

Owning to some perturbation $G_{s,a}^{\epsilon }$ will cause the objective value to decrease, so the above equation must be equal to 0 for $F_{Z\left(s,a\right)}^{-1}$. If $\varphi \left(s,a\right)$ does not equal zero for each $s,a$, since $\varphi \left(s,a\right)$ is arbitrary, it will also cause the above equation to be not equal to 0. Therefore, we obtain

$\begin{array}{l}{\displaystyle {\int }_0^1[p{\left|F_{Z\left(s,a\right)}^{-1}\left(\tau \right)-F_{{\hat{\mathcal{T}}}_{\text{DS}}^{\pi }{\hat{Z}}^k\left(s,a\right)}^{-1}\left(\tau \right)\right|}^{p-1}\text{sign}\left(F_{Z\left(s,a\right)}^{-1}\left(\tau \right)-F_{{\hat{\mathcal{T}}}_{\text{DS}}^{\pi }{\hat{Z}}^k\left(s,a\right)}^{-1}\left(\tau \right)\right)}+\beta c_0\left(s,a\right)\\ +p{\left|F_{Z\left(\hat{s},a\right)}^{-1}\left(\tau \right)-F_{Z\left(s,a\right)}^{-1}\left(\tau \right)\right|}^{p-1}\text{sign}\left(F_{Z\left(\hat{s},a\right)}^{-1}\left(\tau \right)-F_{Z\left(s,a\right)}^{-1}\left(\tau \right)\right)]{\varphi }_{s,a}\left(\tau \right)\text{d}\tau =0.\end{array}$

for all $s,a$. According to the above term is zero for all $\varphi \left(s,a\right)$, we have

$\begin{array}{l}p{\left|F_{Z\left(s,a\right)}^{-1}\left(s,a\right)\left(\tau \right)-F_{{\hat{\mathcal{T}}}_{\text{DS}}^{\pi }{\hat{Z}}^k\left(s,a\right)}^{-1}\left(\tau \right)\right|}^{p-1}\text{sign}\left(F_{Z\left(s,a\right)}^{-1}\left(\tau \right)-F_{{\hat{\mathcal{T}}}_{\text{DS}}^{\pi }{\hat{Z}}^k\left(s,a\right)}^{-1}\left(\tau \right)\right)\\ +\beta c_0\left(s,a\right)+p{\left|F_{Z\left(\hat{s},a\right)}^{-1}\left(\tau \right)-F_{Z\left(s,a\right)}^{-1}\left(\tau \right)\right|}^{p-1}\text{sign}\left(F_{Z\left(\hat{s},a\right)}^{-1}\left(\tau \right)-F_{Z\left(s,a\right)}^{-1}\left(\tau \right)\right)=0.\end{array}$

According to Assumption 3, the above equation can be converted to

$p{\left|F_{Z\left(s,a\right)}^{-1}\left(\tau \right)-F_{{\hat{\mathcal{T}}}_{\text{DS}}^{\pi }{\hat{Z}}^k\left(s,a\right)}^{-1}\left(\tau \right)\right|}^{p-1}\text{sign}\left(F_{Z\left(s,a\right)}^{-1}\left(\tau \right)-F_{{\hat{\mathcal{T}}}_{\text{DS}}^{\pi }{\hat{Z}}^k\left(s,a\right)}^{-1}\left(\tau \right)\right)=\beta c_0\left(s,a\right)\pm {\sigma }^{p-1},$

which holds if and only if

$F_{Z\left(s,a\right)}^{-1}\left(\tau \right)=F_{{\hat{\mathcal{T}}}_{\text{DS}}^{\pi }{\hat{Z}}^k\left(s,a\right)}^{-1}\left(\tau \right)-c\left(s,a\right),$

where $c\left(s,a\right)={\left|\beta p^{-1}{c}_0\left(s,a\right)\pm {\sigma }^{p-1}\right|}^{\frac{1}{p-1}}\cdot \text{sign}\left(c_0\left(s,a\right)\right)$. ≤

Lemma 2. For all $\delta \in {ℝ}_{>0}$, with probability at least $1-\delta$, for any $Z\in \mathcal{Z}$ and $\left(s,a\right)\in \mathcal{D}$, we have

${\Vert F_{{\hat{\mathcal{T}}}^{\pi }Z\left(s,a\right)}-F_{{\mathcal{T}}^{\pi }Z\left(s,a\right)}\Vert }_{\infty }\le \sqrt{\frac{5\left|\mathcal{S}\right|}{n\left(s,a\right)}\text{log}\frac{4\left|\mathcal{S}\right|\left|\mathcal{A}\right|}{\delta }},$(6)

where $n\left(s,a\right)$ represents the number of occurrences of $\left(s,a\right)$ in $\mathcal{D}$.

Proof. Applying the definition of distributional soft Bellman operator to the cumulative density function, we obtain that

$\begin{array}{l}{F}_{{\hat{\mathcal{T}}}_{\text{DS}}^{\pi }Z\left(s,a\right)}\left(x\right)-F_{{\mathcal{T}}_{\text{DS}}^{\pi }Z\left(s,a\right)}\left(x\right)\\ =\underset{s^{\prime },a^{\prime }}{{\displaystyle \sum }}\hat{P}\left(s^{\prime }|s,a\right)\pi \left(a^{\prime }|s^{\prime }\right)F_{\gamma \left[Z\left(s^{\prime },a^{\prime }\right)-c\cdot \log \pi \left(a^{\prime }|s^{\prime }\right)\right]+\hat{R}\left(s,a\right)}\left(x\right)\\ -\underset{s^{\prime },a^{\prime }}{{\displaystyle \sum }}P\left(s^{\prime }|s,a\right)\pi \left(a^{\prime }|s^{\prime }\right)F_{\gamma \left[Z\left(s^{\prime },a^{\prime }\right)-c\cdot \log \pi \left(a^{\prime }|s^{\prime }\right)\right]+R\left(s,a\right)}\left(x\right).\end{array}$

Adding and subtracting ${\displaystyle {\sum }_{s^{\prime },a^{\prime }}\hat{P}}\left(s^{\prime }|s,a\right)\pi \left(a^{\prime }|s^{\prime }\right)F_{\gamma \left[Z\left(s^{\prime },a^{\prime }\right)-c\cdot \log \pi \left(a^{\prime }|s^{\prime }\right)\right]+R\left(s,a\right)}\left(x\right)$ from this expression gives