深度强化学习专栏（三）

Initialization：
Initial a polic π
For s in S:
    For a in A:
        Initial Q(s,a) and Return(s,a)
        c(s,a)=0
For i in range(k):
Generate an episode using π
For each state s appearing in the episode:
G ← the return that follows the first occurrence of s
Q(s_t,a_t)=(c(s_t,a_t)*Q(s_t,a_t)+G)/(c(s_t,a_t)+1)
c(s_t,a_t)++
end for
update policy π(s)=argmaxQ(s,a)
end for

For s in S:
For a in A:
    Initialize Q(s,a)
For i in range(k):
Initialize S
Choose a using policy derived from Q (e.g.ε-greedy)
Repeat (for each step of episode):
Take action a,observe r and s
Choose a^' using policy derived from Q (e.g.ε-greedy)
Q(s, a) ← Q(s, a) + α[R + γQ(s^', a^') − Q(s, a)]
s ←s^'; a ←a^'
until s is terminal
End for

For s in S:
For a in A:
    Initialize Q(s,a)
For i in range(k):
Initialize S
Repeat (for each step of episode):
Choose a using policy derived from Q (e.g.ε-greedy)
Take action a,observe r and s
Q(s, a) ← Q(s, a) + α[R + γmax(Q(s^', a^')) − Q(s, a)]
s ←s^'
until s is terminal
End for

Input: a differentiable policy parameterization π_θ (a|s)
Initialize policy parameter θ
Repeat:
Generate an episode τ: S_0,A_0,R_1,…, S_(T-1), A_(T-1), R_T, following π_θ (a|s)
For each step of the episode t = 0,…,T-1:
G ← return from step t
                  θ← θ+ 〖αγ〗^t G(τ_(t:T) ) ∂/∂θ logπ_θ (a_t |s_t)
    End
Until θ is converged
Output: π_θ

Input: a differentiable policy parameterization π_θ (a|s)
Input: a differentiable state-value parameterization V_∅ (s)
Repeat:
Initialize state s
         ℷ=1
  Repeat:
      On state s, choose an action a=π_θ (a|s)
      Execute a, get reward r and new state s^'
      δ←r+γV_∅ (s^' )-V_∅ (s)
      ∅←∅+βλδ ∂/(∂∅) V_∅ (s) #更新值函数的参数
      θ←θ+αλδ ∂/∂θ logπ_θ (a|s) #更新策略函数的参数
      λ←γλ
      s←s^'
  until s is terminal
until θ is converged
Output: π_θ