A practical way to address the high-dimensional curse is using a parametric function to estimate the value function. In this paper, we focus on extending Q ( σ , λ ) with linear function approximation. Since the divergence of semi-gradient with multi-step bootstrapping for off-policy learning are well-documented in the existing literature (e.g., [3][8]), which could also happen in semi-gradient Q ( σ , λ ) . To propose a convergent gradient-based algorithm, we derive the GQ ( σ , λ ) algorithm via the mean square projected Bellman error (MSPBE) objective function [9], that inspired by weight-duplication trick (also known as “two-timescale stochastic approximation”) [9][10]. When σ ranges from 0 to 1, GQ ( σ , λ ) ranges from gradient Tree Backup ( λ ) ( TB ( λ ) ) to GQ ( λ ) [11], i.e., our GQ ( σ , λ ) unifies gradient Tree Backup ( λ ) and GQ ( λ ) .

Although GQ ( σ , λ ) | σ = 0 is a natural algorithm to extend TB ( λ ) with linear function approximation, to the best of our knowledge, the update rule of GQ ( σ , λ ) | σ = 0 has not been proposed in the existing literatures. It is worth to notice that Touati et al., [12] have proposed another version of gradient TB ( λ ) ( GTB ( λ ) ), which is different from the proposed GQ ( σ , λ ) | σ = 0 , we have clarified this point in Remark 3.

Then, we provide the convergence analysis of the proposed GQ ( σ , λ ) . Theorem 1 shows that GQ ( σ , λ ) converges to its TD fixed-point with probability one. Additionally, Theorem 1 illustrates the structure of such TD fixed-point: it is the global asymptotically stable equilibrium of its corresponding ordinary differential equation (ODE). For more discussion, see Remark 4. Furthermore, Theorem 2 shows that GQ ( σ , λ ) converges to an arbitrarily small neighborhood of the optimal solution with probability one.

Finally, our experiments show that when σ ranges from 0 to 1, GQ ( σ , λ ) achieves the best performance of off-policy evaluation or control within a certain σ ∈ ( 0 , 1 ) , neither σ = 0 , nor σ = 1 , which implies that with a certain value σ ∈ ( 0 , 1 ) , GQ ( σ , λ ) creates a mixture between GQ ( λ ) and gradient TB ( λ ) reaches a better performance than the extreme ends ( σ = 0 and σ = 1 ).

Reinforcement learning (RL) [3] is often formalized as Markov decision processes (MDP) that considers a tuple ℳ = ( S , A , P , R , γ ) ; S is a set with finite states, A is a set with finite actions; P : S × A × S → [ 0 , 1 ] , p s s ′ a = P ( S t = s ′ | S t − 1 = s , A t − 1 = a ) is the probability of state transition from s to s ′ under playing the action a ; R ( ⋅ , ⋅ ) : S × A → ℝ 1 is the expected reward function; γ ∈ ( 0 , 1 ) .

A policy π is a probability distribution on S × A , and π ( a | s ) denotes the probability of playing a in state s . Let { S t , A t , R t + 1 } t ≥ 0 be generated by π , its state-action value function is: q π ( s , a ) = E π [ ∑ t = 0 ∞ γ t R t + 1 | S 0 = s , A 0 = a ] , where E π [ ⋅ | ⋅ ] is conditional expectation on the actions selected according to π . Let ℬ π : ℝ | S | → ℝ | S | denote Bellman operator with respect to policy π :

where P π ∈ ℝ | S | × | S | , R π ∈ ℝ | S | × | A | , their corresponding elements are: [ P π ] s , s ′ = ∑ a ∈ A π ( a | s ) p s s ′ a , [ R π ] s , a = R ( s , a ) . It is well-known that q π is the unique fixed point of ℬ π , i.e., ℬ π q π = q π , which is known as Bellman equation.

Let us consider the trajectory τ = : { S t , A t , R t + 1 } t ≥ 0 generated by the behavior policy μ , where A t ~ μ ( ⋅ | S t ) , S t + 1 ~ P ( ⋅ | S t , A t ) . Off-policy evaluation is the task to estimate the value function of the target policy π via the data that is generated by the behavior policy μ , where μ ≠ π .

Assumption 1 (Ergodicity).The Markov chain induced by behavior policy μ is ergodic, i.e., there exists a stationary distribution ξ ( ⋅ , ⋅ ) over S × A : for ∀ ( S 0 , A 0 ) ,

The ergodicity of behavior policy μ is a standard assumption in off-policy learning [3], and it implies each-action pair is visited under this behavior policy μ . We use Ξ to denote a diagonal matrix whose diagonal element is ξ ( s , a ) , i.e., Ξ = diag { ⋯ , ξ ( s , a ) , ⋯ } .

TD learning updates value function as follows, ∀ t ≥ 0 ,

where Q ( ⋅ , ⋅ ) is an estimator of q π , α t is step-size and δ t is TD error. Let Q t = : Q ( S t , A t ) , if

then update (3) is Sarsa [4]. If δ t is expected TD error:

then update (3) is Expected Sarsa [4].

The λ -return is an average contains all the n -step returns by weighting proportionally to λ n − 1 , where λ ∈ [ 0 , 1 ] . In this paper, we mainly consider two classic λ -return: Tree Backup ( λ ) ( TB ( λ ) ) and Expected Sarsa ( λ ) .

TreeBackup( λ ). For each pair ( S t , A t ) in the trajectory τ , TB ( λ ) [5] estimates q π ( S t , A t ) by

where δ k ES is expected TD error. Precup et al., [5] have proved the iteration (6) converges to q π with probability one under some certain conditions.

ExpectedSarsa( λ ). Sutton and Barto [3] have proposed a multi-step TD learning extends Expected Sarsa to λ -return version: for each t ≥ 0 ,

For the convenience, in the following paragraph, we consider the following notations,

In this section, we review an approach to unify TB ( λ ) and Expected Sarsa ( λ ) .

Q ( σ ) Algorithm. Recently, De Asis et al., [2] propose Q ( σ ) unifies multi-step Sarsa and multi-step TB ( 0 ) . Concretely, according to a mixed TD error δ t π , σ :

De Asis et al., [2] construct a multi-step estimator:

where σ ∈ [ 0 , 1 ] is sampling parameter. When σ ranges from 0 to 1, the update (9) ranges from multi-step TB ( 0 ) to multi-step Sarsa. Experimental results show that a certain σ ∈ ( 0 , 1 ) results in a mixture of TB ( 0 ) and Sarsa performs better than both σ = 0 and σ = 1 [2], which implies unifying some seemingly disparate algorithmic ideas can create better performing algorithms.

Off-Policy Q ( σ , λ ) . Later, De Asis, [7] proposes a multi-step returns as follows,

where

When σ ranges from 0 to 1, the estimator (10) ranges from TB ( λ ) (6) to Expected Sarsa ( λ ) (7).

We introduce a λ -operator ℬ σ , λ π , μ ( ⋅ ) : ℝ | S | × | A | → ℝ | S | × | A | that is a high level view of the λ -return (10),

where ℬ π is Bellman operator (1), P π , μ ∈ ℝ | S | × | S | , and whose elements are: [ P π , μ ] s , s ′ = ∑ a ∈ A π ( a | s ) μ ( a | s ) p s s ′ a .

Remark 1.Yang et al. [6]propose another version of Q ( σ , λ ) algorithm that extends Q ( σ ) (9) with eligibility trace:

It is noteworthy that at one extreme end σ = 1 , both Q ( σ ) (9) and G ˜ t σ , λ (13) reduce to on-policy learning. Particularly, [6]prove that the performance of G ˜ t σ , λ for off-policy evaluation is determined by parameter σ :

where C is a positive constant never reaches0 no matter how we choose the starting time t . The upper error bound of(14) illustrates the capacity of G ˜ t σ , λ for off-policy evaluation decays monotonously when σ ranges from0to1. At the extreme end σ = 1 , G ˜ t σ , λ achieves the worst performance of off-policy evaluation. We think this is a natural result since G ˜ t σ , λ | σ = 1 is an on-policy algorithm exactly. Thus, in this paper, we mainly concern (10) for off-policy learning.

TD learning (3) requires a very huge table to store the estimate value function Q ( ⋅ , ⋅ ) when | S | is very large, which implies tabular TD learning is considerably expensive for high-dimensional RL. We often use a parametric function Q θ ( ⋅ , ⋅ ) to approximate

where θ ∈ ℝ p is the parameter need to be learned, ϕ ( s , a ) = ( φ 1 ( s , a ) , φ 2 ( s , a ) , ⋯ , φ p ( s , a ) ) ⊤ , and each φ i : S × A → ℝ . Furthermore, Q θ can be rewritten as a version of matrix

where Φ is a matrix whose rows are the state-action feature vectors ϕ ⊤ ( s , a ) .

Recall τ = { ( S t , A t , R t + 1 ) } t ≥ 0 is generated by behavior policy μ , let ϕ t = ϕ ( S t , A t ) , we define semi-gradient Q ( σ , λ ) as follows:

where

is an off-line estimator of value function according to Equation (10), TD error δ k ES ( θ t ) = R k + 1 + γ E π [ θ t ⊤ ϕ ( S k + 1 , ⋅ ) ] − θ t ⊤ ϕ k , and α t is step-size. Let

Then update (17) can be rewritten as follows,

Furthermore, we have

Now, we only briefly discuss the divergence lies in the iteration (17). Under the conditions of Proposition 4.8 presented by Bertsekas and Tsitsiklis [14], { θ t } t ≥ 0 (17) converges to a certain point if and only if A σ is a negative matrix. Furthermore, if iteration (17) converges, then it converges to its unique TD fixed point θ ⋆ that satisfes

Unfortunately, since the steady state-action distribution doesn’t match the transition probability during off-policy learning, we can not guarantee the negative definiteness of A σ , thus { θ t } t ≥ 0 may diverge. To clarify this point, we use the classic counterexample [12] to show the divergence of semi-gradient TD algorithms (17) for off-policy learning.

Figure 1. Counterexample from [12]: Two-State MDP.

For the MDP in Figure 1, the behavior policy μ ( right | ⋅ ) = 0.5 , and target policy π ( right | ⋅ ) = 1 . We assign the features { ( 1 , 0 ) ⊤ , ( 2 , 0 ) ⊤ , ( 0 , 1 ) ⊤ , ( 0 , 2 ) ⊤ } to the state-action pairs { ( s 1 , right ) , ( s 2 , right ) , ( s 1 , left ) , ( s 2 , left ) } , From the dynamic transition shown in Figure 1, we have

Then, according to (21), we have

and the eigenvalues of A are: 6 ( 2 − σ ) γ − γ λ − 5 ( 2 − σ ) 2 ( 1 − γ λ ) and − 5 ( 1 − σ 2 ) . For any initial θ 0 = ( θ 0 , 1 , θ 0 , 2 ) ⊤ , let E [ θ t + 1 ] = : ( θ t + 1 , 1 , θ t + 1 , 2 ) ⊤ , according to (20), the first component of the term E [ θ t + 1 | θ t ] is:

For any λ ∈ ( 0 , 1 ) , if γ ∈ ( 10 − 5 σ 12 − 6 σ − λ , 1 ) , then 6 ( 2 − σ ) γ − γ λ − 5 ( 2 − σ ) 2 ( 1 − γ λ ) is a positive scalar, which implies A σ can not be a negative matrix. Furthermore, if step size α t : ∑ i ≥ 0 α t = ∞ , we have

Equation (25) is a direct result of the following conclusion that could be found in any calculus textbook. Let p i = 1 + a i , where a i > 0 , if ∑ i = 1 ∞ a i = + ∞ , then ∏ i = 1 ∞ p i = ∏ i = 1 ∞ ( 1 + a i ) = + ∞ , which implies the way (17) to extend Q ( σ ) with linear function approximation via off-line estimate is unstable for off-policy learning.

We need some additional assumptions to present the convergent of GQ ( σ , λ ) , those assumptions are widely used in reinforcement learning [13][17]-[19].

Assumption 2 (Diminishing Step-size).The positive sequences { α t } t ≥ 0 , { β t } t ≥ 0 satisfy the following conditions with probability one,

Assumption 3.The features { ϕ t } t ≥ 0 is uniformly bounded by ϕ max . The reward function is uniformly bounded by R max . The importance sampling ρ t = π ( A t | S t ) μ ( A t | S t ) is uniformly bounded by ρ max .

Assumption 4 (Solvability of Problem).The matrix A σ is non-singular and rank ( Φ ) = p .

Assumption 4 requires the non-singular matrix A σ , which implies the optimal parameter θ ⋆ = − A − 1 b is well defined. The feature matrix Φ has linearly independent columns implies the matrix M = Φ ⊤ Ξ Φ is positive defined.

Theorem 1 (Convergence ofAlgorithm 1).Under Assumption 1-4, we consider the iteration { ( θ t , ω t ) } t ≥ 0 that is generated according to (30)-(31). The step-size α t , β t satisfy Assumption2 and η t = β t α t → 0 , as t → ∞ . We define two functions G ( θ ) , H ( ω , θ ) as follows,

Then ( θ t , ω t ) → w . p .1 ( θ ⋆ , ω ⋆ ) , as t → ∞ , where ( θ ⋆ , ω ⋆ ) is the unique global asymptotically stable equilibrium with respect to the following ODE correspondingly:

where θ ( t ) , ω ( t ) ∈ ℝ p are the functions are defined on continuous time ( 0 , ∞ ) .

Proof. We provide its proof in Appendix B. □

Remark 4 (TD-Fixed Point of GQ ( σ , λ ) ).Theorem1 illustrates that the sequence { ( θ t , ω t ) } t ≥ 0 generated by GQ ( σ , λ ) converges to the global asymptotically stable equilibrium of its corresponding ODE (35). Furthermore, from the details of the proof inAppendix B, we know since A σ is invertible and M − 1 is positive definite, then A σ ⊤ M − 1 A σ is also positive defined. Because the ODE uses the negative MSPBE-gradient direction, the Jacobian at the equilibrium is − A σ ⊤ M − 1 A σ , whose eigenvalues have negative real parts.Sothe following ODE

has a unique global asymptotically stable equilibrium θ ⋆ satisfies the equation

which implies the global asymptotically stable equilibrium θ ⋆ of the ODE θ ˙ ( t ) = G ( θ ( t ) ) is also the TD fixed point of (23). That means θ t converges to its TD fixed point:

Remark 5 (Unification of TD-Fixed Point).If σ = 0 , the matrix A σ is reduced to

which implies GQ ( σ , λ ) | σ = 0 converges to the TD-fixed point of GTB ( λ ) [12], i.e., GQ ( σ , λ ) | σ = 0 shares the same TD-fixed point of GTB ( λ ) [12] as follows

If σ = 1 , then the key matrix A σ is reduced to

which illustrates the TD-fixed point of GQ ( σ , λ ) | σ = 1 (i.e., GQ ( λ ) ) algorithm as follows

Theorem 1 presents an asymptotic result, which holds only in the limit as the number of iterations increases to infinity. Now, we present a result shows the distance between θ t and the optimal solution θ ⋆ convergence to 0 in probability. For any T > 0 , and t ≥ 0 , we introduce a notation

to denote the last iteration before the sum of step-size α i between it and the t -th iteration exceeds T . Since we consider the Assumption 2, the notation κ ( t ; T ) is well-defined.

Theorem 2.Under Assumption 1-4, we consider the iteration { ( θ t , ω t ) } t ≥ 0 generated by (30)-(31). The step-size α t , β t satisfy Assumption2. Moreover, for the integers m t → ∞ , m ′ t → ∞ , as t → ∞ ,

Then there exists a sequence of positive numbers T t → ∞ (as t → ∞ ) such that for any ϵ > 0 ,

where we use N ϵ ( θ ⋆ ) = { θ : ‖ θ − θ ⋆ ‖ 2 ≤ ϵ } to denote the ϵ -neighborhood of θ ⋆ .

Proof. We provide its proof in Appendix C. □

Remark 6Theorem2 shows that as t → ∞ , the sequence { θ i } i = t κ ( t , T t ) collects to an arbitrarily small neighborhood of the optimal solution θ ⋆ with probability one, i.e.,

In fact, the results (38) implies

then we know