Chapter3 Markov Decision Processes(MDP)

参考了《Reinforcement Learning: An Introduction》和
David Silver强化学习公开课，
这一章主要来自David Silver的ppt，建议直接看ppt，我只把容易犯错的地方点出来了

---

马尔科夫过程是强化学习的基础

---

Finite Markov Decision Processes

Markov property

A state St$S_t$ is Markov if and only of

P[St+1|St]=P[St+1|S1,⋯,St]$$\mathbb{P}[S_{t+1}|S_t]=\mathbb{P}[S_{t+1}|S_1,\cdots ,S_t]$$

• The state captures all relevant information from the history
• Once the state is know, the history may be thrown away
• i.e. The state is a sufficient statistic of the future

A Markov process is a memoryless random process, i.e. a sequence of random states S1,S2,⋯$S_1,S_2,\cdots$ with the Markov property.
Markov Process

A Markov Process (or Markov Chain) is a tuple ⟨S,P⟩$⟨S,P⟩$

• S is a (finite) set of states
• P is a state transition probability matrix, Pss′=P[St+1=s′|St=s]$P_{s{s}^{\mathrm{\prime }}}=\mathbb{P}[S_{t+1}=s^{\mathrm{\prime }}|S_t=s]$

A Markov reward process is a Markov chain with values.
Markov Reward Process

A Markov Process (or Markov Chain) is a tuple ⟨S,P,R,γ⟩$⟨S,P,R,\gamma ⟩$

• S is a (finite) set of states
• P is a state transition probability matrix, Pss′=P[St+1=s′|St=s]$P_{s{s}^{\mathrm{\prime }}}=\mathbb{P}[S_{t+1}=s^{\mathrm{\prime }}|S_t=s]$
• R is a reward function, Rs=E[Rt+1|St=s]$\text{R is a reward function, }{R}_s=E[R_{t+1}|S_t=s]$
• γ is a discount factor, γ∈[0,1]$\gamma \text{ is a discount factor, }\gamma \in [0,1]$

注意这里Pss′$P_{s{s}^{\mathrm{\prime }}}$的定义，是指从状态s$s$到s′$s^{\mathrm{\prime }}$的概率

后面常因为名字(return)忘记这个的定义，跟上面的单个Reward不一样
Return

The return Gt$G_t$ is the total discounted reward from time-step t.

Gt=Rt+1+γRt+2+⋯=∑k=0∞γkRt+k+1$$G_t=R_{t+1}+\gamma R_{t+2}+\cdots =\sum _{k=0}^{\mathrm{\infty }}{\gamma }^k{R}_{t+k+1}$$

• The discount γ∈[0,1]$\gamma \in [0,1]$ is the present value of future rewards
• The value of receiving reward R after k+1 time-steps is γkR${\gamma }^{k}R$
  
  • γ$\gamma$ close to 0 leads to “myopic(近视)” evaluation
  • γ$\gamma$ close to 1 leads to “far-sighted(远见)” evaluation
    后面提到的很多方法都是看的很远(远见)的

Value Function

The state value function v(s) of an MRP$\text{MRP}$ is the expected return starting form state s

v(s)=E[Gt|St=s]$$v(s)=\mathbb{E}[G_t|S_t=s]$$

确实有必要看一下MRP的Bellman Equation，并与MDP对比。在MRP中没有考虑任何关于action的事情。因为MDP才是强化学习的主角，所以不看David Silver的ppt中的MRP实例了，容易对后面MDP的理解造成误解。
简单看一下Bellman Equation

v(s)=E[Gt|St=s]=E[Rt+1+γv(St+1)|St=s]$$\begin{array}{rl}v(s)& =\mathbb{E}[G_t|S_t=s]\\ & =\mathbb{E}[R_{t+1}+\gamma v(S_{t+1})|S_t=s]\end{array}$$

MRP的状态转换，没有任何action的影响，我们在后面MDP中会考虑actions的影响

v(s)=Rs+γ∑s′∈SPss′v(s′)$$v(s)=R_s+\gamma \sum _{s^{\mathrm{\prime }}\in S}{P}_{s{s}^{\mathrm{\prime }}}v(s^{\mathrm{\prime }})$$

其实观察上式，上面计算的是动态规划，而注意到Bellman Equation又称为动态规划方程，上面的计算就很容易理解了

---

A Markov decision process (MDP) is a Markov reward process with decisions. It is an environment in which all states are Markov.
Markov Decision Process

A Markov Process (or Markov Chain) is a tuple ⟨S,A,P,R,γ⟩$⟨S,A,P,R,\gamma ⟩$

• S is a (finite) set of states
• A is finite set of actions
• P is a state transition probability matrix, Pass′=P[St+1=s′|St=s,At=a]$P_{s{s}^{\mathrm{\prime }}}^{\text{a}}=\mathbb{P}[S_{t+1}=s^{\mathrm{\prime }}|S_t=s,A_t=\text{a}]$
• R is a reward function, Ras=E[Rt+1|St=s,At=a]$R_s^{\text{a}}=E[R_{t+1}|S_t=s,A_t=\text{a}]$
• γ$\gamma$ is a discount factor, γ∈[0,1]$\gamma \in [0,1]$

注意与上面MRP的区别，这里的黑点是执行一个action之后到达的中间状态，后面用q(s,a)$q(s,a)$来定义此状态，黑点到达后面的状态s′$s^{\mathrm{\prime }}$的概率就是上面MDP中定义的那个Pass′=P[St+1=s′|St=s,At=a]$P_{s{s}^{\mathrm{\prime }}}^a=\mathbb{P}[S_{t+1}=s^{\mathrm{\prime }}|S_t=s,A_t=a]$

Policy

A policy π$\pi$ is a distribution over actions given states,

π(a|s)=P[At=a|St=s]$$\pi (a|s)=\mathbb{P}[A_t=a|S_t=s]$$

• A policy fully defines the behaviour of an agent
• MDP policies depend on the current state (not the history)
• i.e. Policies are stationary (time-independent), At∼π(⋅|St),∀t>0$A_t\sim \pi (\cdot |S_t),\mathrm{\forall }t>0$
• Given an MDP M=⟨S,A,P,R,γ⟩$M=⟨S,A,P,R,\gamma ⟩$ and a policy π$\pi$
• The state sequence S1,S2,⋯$S_1,S_2,\cdots$ is a Markov reward process ⟨S,Pπ⟩$⟨S,P^{\pi }⟩$
• The state and reward sequence S1,R2,S2,⋯$S_1,R_2,S_2,\cdots$ is a Markov reward process ⟨S,Pπ,Rπ,γ⟩$⟨S,P^{\pi },R^{\pi },\gamma ⟩$
• where
  
  Pπs,s′=∑a∈Aπ(a|s)Pass′Rπs=∑a∈Aπ(a|s)Ras$$\begin{gathered} P_{s,s^{\mathrm{\prime }}}^{\pi }=\sum _{a\in A}\pi (a|s)P_{s{s}^{\mathrm{\prime }}}^a \\ {R}_s^{\pi }=\sum _{a\in A}\pi (a|s)R_s^a \end{gathered}$$

要特别注意policy的distribution的定义，因为在后面讲的off-policy方法的概念中，生成样本的policy和目标policy是不同的

Value Function这个是针对MDP的

The state-value function vπ(s)$v_{\pi }(s)$ of an MDP is the expected return starting from state s$s$, and then following policy π$\pi$

vπ(s)=Eπ[Gt|St=s]$$v_{\pi }(s)={\mathbb{E}}_{\pi }[G_t|S_t=s]$$

The action-value function qπ(s,a)$q_{\pi }(s,a)$ is the expected return
starting from state s$s$, taking action a$a$, and then following policy π$\pi$

qπ(s|a)=Eπ[Gt|St=s,At=a]$$q_{\pi }(s|a)={\mathbb{E}}_{\pi }[G_t|S_t=s,A_t=a]$$

Bellman Expectation Equation for Vπ$V^{\pi }$

vπ(s)=∑a∈Aπ(a|s)qπ(s,a)$$v_{\pi }(s)=\sum _{a\in A}\pi (a|s)q_{\pi }(s,a)$$

Bellman Expectation Equation for Qπ$Q^{\pi }$

qπ(s,a)=Ras+γ∑s′∈SPass′vπ(s′)$$q_{\pi }(s,a)=R_s^a+\gamma \sum _{s^{\mathrm{\prime }}\in S}{P}_{s{s}^{\mathrm{\prime }}}^a{v}_{\pi }(s^{\mathrm{\prime }})$$

vπ(s)=∑a∈Aπ(a|s)(Ras+γ∑s′∈SPass′vπ(s′))$$v_{\pi }(s)=\sum _{a\in A}\pi (a|s)(R_s^a+\gamma \sum _{s^{\mathrm{\prime }}\in S}{P}_{s{s}^{\mathrm{\prime }}}^a{v}_{\pi }(s^{\mathrm{\prime }}))$$

qπ(s,a)=Ras+γ∑s′∈SPass′∑a′∈Aπ(a′|s′)qπ(s′,a′)$$q_{\pi }(s,a)=R_s^a+\gamma \sum _{s^{\mathrm{\prime }}\in S}{P}_{s{s}^{\mathrm{\prime }}}^a\sum _{a^{\mathrm{\prime }}\in A}\pi (a^{\mathrm{\prime }}|s^{\mathrm{\prime }})q_{\pi }(s^{\mathrm{\prime }},a^{\mathrm{\prime }})$$

Optimal Value Function

The optimal state-value function v∗(s)$v_{\ast }(s)$ is the maximum value function over all policies

v∗(s)=maxπvπ(s)$$v_{\ast }(s)=\underset{\pi }{max}{v}_{\pi }(s)$$

The optimal action-value function q∗(s,a)$q_{\ast }(s,a)$ is the maximum action-value function over all policies

q∗(s,a)=maxπqπ(s,a)$$q_{\ast }(s,a)=\underset{\pi }{max}{q}_{\pi }(s,a)$$

只要知道了q∗$q_{\ast }$问题就解决了，比知道v∗$v_{\ast }$更方便。还有注意的是，上面是在所有的π$\pi$(policy)中选择使得 q$q$ 最大的π$\pi$(policy)，这就是值给出了最佳policy的概念，当然是没有很直接的办法得到结果的，后面将针对上述问题介绍各种逼近的方法

Optimal Policy
Dene a partial ordering over policies

π≥π′ if vπ(s)≥vπ′(s),∀s$$\pi \ge {\pi }^{\mathrm{\prime }}\text{if}{v}_{\pi }(s)\ge v_{{\pi }^{\mathrm{\prime }}}(s),\mathrm{\forall }s$$

Finding an Optimal Policy
An optimal policy can be found by maximising over q∗(s,a)$q_{\ast }(s,a)$,

π∗(a|s)={10if a = argmaxa∈Aq∗(s,a)otherwise$${\pi }_{\ast }(a|s)=\{\begin{array}{ll}1& \text{if a = }\underset{a\in A}{argmax}{q}_{\ast }(s,a)\\ 0& \text{otherwise}\end{array}$$

如果我们知道了q∗(s,a)$q_{\ast }(s,a)$，那么我就可以马上得到optimal policy

Optimal Bellman Expectation Equation

vπ(s)≐Eπ[Gt|St=s]=Eπ[∑k=0∞γkRt+k+1|St=s]=∑aπ(a|s)∑s′∑rp(s′,r|s,a)[r+γEπ[Gt+1|St+1=s′]]=∑aπ(a|s)∑s′,rp(s′,r|s,a)[r+γvπ(s′)], for all s∈S$$\begin{array}{rl}{v}_{\pi }(s)& \doteq {\mathbb{E}}_{\pi }[G_t|S_t=s]\\ & ={\mathbb{E}}_{\pi }[\sum _{k=0}^{\mathrm{\infty }}{\gamma }^k{R}_{t+k+1}|S_t=s]\\ & =\sum _a\pi (a|s)\sum _{s^{\mathrm{\prime }}}\sum _{r}p(s^{\mathrm{\prime }},r|s,a)[r+\gamma {\mathbb{E}}_{\pi }[G_{t+1}|S_{t+1}=s^{\mathrm{\prime }}]]\\ & =\sum _a\pi (a|s)\sum _{s^{\mathrm{\prime }},r}p(s^{\mathrm{\prime }},r|s,a)[r+\gamma v_{\pi }(s^{\mathrm{\prime }})],\text{ for all }s\in S\end{array}$$

The Agent-Environment Interface

• The learner and decision maker is called the agent.
• The thing it interacts with, comprising everything outside the agent, is called the environment.

MDP和agent一起生成的sequence或者trajectory

S0,A0,R1,S1,A1,R2,S2,A2,R3,⋯$$S_0,A_0,R_1,S_1,A_1,R_2,S_2,A_2,R_3,\cdots$$

以下函数定义了MDP的动态性，agent处于某个状态s，在该状态下采取行动a，然后到达状态s′$s^{\mathrm{\prime }}$，并获得奖励r。这个公式是MDP的关键。这个四参数的函数可以推导出任何东西

p(s′,r|s,a)≐Pr{St=s′,Rt=r|St−1=s,At−1=a}$$p(s^{\mathrm{\prime }},r|s,a)\doteq Pr\{S_t=s^{\mathrm{\prime }},R_t=r|S_{t-1}=s,A_{t-1}=a\}$$

for all s′$s^{\mathrm{\prime }}$, s∈S$s\in S$, r∈R$r\in R$, and a∈A(s)$a\in A(s)$

其中有

∑s′∈S∑r∈Rp(s′,r|s,a)=1, for all s∈S, a∈A(s)$$\sum _{s\mathrm{\prime }\in S}\sum _{r\in R}p(s^{\mathrm{\prime }},r|s,a)=1,\text{ for all }s\in S\text{, }a\in A(s)$$

3.2 Goals and Rewards

agent的目的就是最大化它收到的全部rewards

3.5 Policies and Value Functions

state-value function for policy π$\pi$

vπ(s)≐Eπ[Gt|St=s]=Eπ[∑k=0∞γkRt+k+1|St=s]=∑aπ(a|s)∑s′∑rp(s′,r|s,a)[r+γEπ[Gt+1|St+1=s′]]=∑aπ(a|s)∑s′,rp(s′,r|s,a)[r+γvπ(s′)], for all s∈S$$\begin{array}{rl}{v}_{\pi }(s)& \doteq E_{\pi }[G_t|S_t=s]\\ & =E_{\pi }[\sum _{k=0}^{\mathrm{\infty }}{\gamma }^k{R}_{t+k+1}|S_t=s]\\ & =\sum _a\pi (a|s)\sum _{s^{\mathrm{\prime }}}\sum _{r}p(s^{\mathrm{\prime }},r|s,a)[r+\gamma E_{\pi }[G_{t+1}|S_{t+1}=s^{\mathrm{\prime }}]]\\ & =\sum _a\pi (a|s)\sum _{s^{\mathrm{\prime }},r}p(s^{\mathrm{\prime }},r|s,a)[r+\gamma v_{\pi }(s^{\mathrm{\prime }})],\text{ for all }s\in S\end{array}$$

action-value function for policy π$\pi$

qπ(s,a)≐Eπ[Gt|St=s,At=a]=Eπ[∑k=0∞γkRt+k+1|St=s,At=a]$$q_{\pi }(s,a)\doteq E_{\pi }[G_t|S_t=s,A_t=a]=E_{\pi }[\sum _{k=0}^{\mathrm{\infty }}{\gamma }^k{R}_{t+k+1}|S_t=s,A_t=a]$$

对于任何policy π$\pi$和任何状态s$s$，state-value和其可能的后继状态的state-value之间存在以下一致性条件

3.6 Optimal Policies and Optimal Value Functions

optimal state-value function

v∗(s)≐maxπvπ(s)$$v_{\ast }(s)\doteq \underset{\pi }{max}{v}_{\pi }(s)$$

optimal action-value function

q∗(s,a)≐maxπqπ(s,a)$$q_{\ast }(s,a)\doteq \underset{\pi }{max}{q}_{\pi }(s,a)$$

写出关于v∗$v_{\ast }$的q∗$q_{\ast }$

q∗(s,a)=E[Rt+1+γvπ(St+1)|St=s,At=a]$$q_{\ast }(s,a)=E[R_{t+1}+\gamma v_{\pi }(S_{t+1})|S_t=s,A_t=a]$$

Bellman optimality equation

v∗(s)=maxa∈A(s)qπ∗$$v_{\ast }(s)=\underset{a\in A(s)}{max}{q}_{{\pi }_{\ast }}$$

---

v∗(s)=maxa∈A(s)qπ∗(s,a)=maxaEπ∗[Gt|St=s,At=a]=maxaEπ∗[Rt+1+γGt+1|St=s,At=a]=maxaE[Rt+1+γv∗(St+1)|St=s,At=a]=maxa∑s′,rp(s′,r|s,a)[r+γv∗(s′)]$$\begin{array}{rl}{v}_{\ast }(s)& =\underset{a\in A(s)}{max}{q}_{{\pi }_{\ast }}(s,a)\\ & =\underset{a}{max}{E}_{{\pi }_{\ast }}[G_t|S_t=s,A_t=a]\\ & =\underset{a}{max}{E}_{{\pi }_{\ast }}[R_{t+1}+\gamma G_{t+1}|S_t=s,A_t=a]\\ & =\underset{a}{max}E[R_{t+1}+\gamma v_{\ast }(S_{t+1})|S_t=s,A_t=a]\\ & =\underset{a}{max}\sum _{s^{\mathrm{\prime }},r}p(s^{\mathrm{\prime }},r|s,a)[r+\gamma v_{\ast }(s^{\mathrm{\prime }})]\end{array}$$

---

q∗(s,a)=E[Rt+1+γmaxa′q∗(St+1,a′)|St=s,At=a]=∑s′,rp(s′,r|s,a)[r+γmaxa′q∗(s′,a′)]$$\begin{array}{rl}{q}_{\ast }(s,a)& =E[R_{t+1}+\gamma \underset{a^{\mathrm{\prime }}}{max}{q}_{\ast }(S_{t+1},a^{\mathrm{\prime }})|S_t=s,A_t=a]\\ & =\sum _{s^{\mathrm{\prime }},r}p(s^{\mathrm{\prime }},r|s,a)[r+\gamma \underset{a^{\mathrm{\prime }}}{max}{q}_{\ast }(s^{\mathrm{\prime }},a^{\mathrm{\prime }})]\end{array}$$