前文是一些针对IRL，IL综述性的解释，后文是针对《Generative adversarial imitation learning》文章的理解及公式的推导。
通过深度强化学习，我们能够让机器人针对一个任务实现从0到1的学习，但是需要我们定义出reward函数，在很多复杂任务，例如无人驾驶中，很难根据状态特征来建立一个科学合理的reward。
人类学习新东西有一个重要的方法就是模仿学习，通过观察别人的动作来模仿学习，不需要知道任务的reward函数。模仿学习就是希望机器能够通过观察模仿专家的行为来进行学习。
OpenAI，DeepMind，Google Brain目前都在向这方面发展。
[1] Model-Free Imitation Learning with Policy Optimization, OpenAI, 2016
[2] Generative Adversarial Imitation Learning, OpenAI, 2016
[3] One-Shot Imitation Learning, OpenAI, 2017
[4] Third-Person Imitation Learning, OpenAI, 2017
[5] Learning human behaviors from motion capture by adversarial imitation, DeepMind, 2017
[6] Robust Imitation of Diverse Behaviors, DeepMind, 2017
[7] Unsupervised Perceptual Rewards for Imitation Learning, Google Brain, 2017
[8] Time-Contrastive Networks: Self-Supervised Learning from Multi-View Observation, Google Brain, 2017
[9] Imitation from Observation/ Learning to Imitate Behaviors from Raw Video via Context Translation, OpenAI, 2017
[10] One Shot Visual Imitation Learning, OpenAI, 2017
从给定的专家轨迹中进行学习。
机器在学习过程中能够跟环境交互，到那时不能直接获得reward。
在任务中很难定义合理的reward（自动驾驶中撞人reward，撞车reward，红绿灯reward），人工定义的reward可能会导致失控行为（让agent考试，目标为考100分，但是reward可能通过作弊的方式）。
三种方法：
行为克隆
逆向强化学习
生成对抗模仿学习（GAN for Imitation Learning）
GAIL能够直接从专家轨迹中学得策略，绕过很多IRL的中间步骤。
假定cost function的集合为C \mathcal{C} C π E \pi_E π E ψ \psi ψ I R L ψ ( π E ) = arg m a x c ∈ C − ψ ( c ) + ( min π ∈ Π − H ( π ) + E π [ c ( s , a ) ] ) − E π E [ c ( s , a ) ]
\rm{IRL}_{\psi}(\pi_E) = \arg max_{c\in\mathcal{C}} -\psi(c)+(\min_{\pi \in \Pi}- \it {H}(\pi) + \mathbb E_\pi[c(s,a)]) - \mathbb E_{\pi_E}[c(s,a)]
I R L ψ ( π E ) = arg m a x c ∈ C − ψ ( c ) + ( π ∈ Π min − H ( π ) + E π [ c ( s , a ) ] ) − E π E [ c ( s , a ) ] E π [ c ( s , a ) ] = E [ ∑ t = 0 ∞ γ t c ( s t , a t ) ] \mathbb E_\pi[c(s,a)]=\mathbb E[\sum\limits_{t=0}^\infty \gamma^tc(s_t,a_t)] E π [ c ( s , a ) ] = E [ t = 0 ∑ ∞ γ t c ( s t , a t ) ] H ( π ) = E π [ − log π ( a ∣ s ) ] H(\pi)=\mathbb E_\pi[-\log \pi(a|s)] H ( π ) = E π [ − log π ( a ∣ s ) ] γ \gamma γ R L ( c ) = arg m i n π ∈ Π − H ( π ) + E π [ c ( s , a ) ]
\rm{RL}(c) = \arg min_{\pi \in \Pi} -\it H(\pi)+\mathbb E_\pi
[c(s,a)]
R L ( c ) = arg m i n π ∈ Π − H ( π ) + E π [ c ( s , a ) ]
对于一个策略π \pi π ρ π : S × A → R \rho_\pi:\mathcal{S}\times\mathcal{A}\to \mathbb R ρ π : S × A → R ρ π ( s , a ) = π ( a ∣ s ) ∑ t = 0 ∞ γ t P ( s t = s ∣ π )
\rho_\pi(s,a) = \pi(a|s)\sum\limits_{t=0}^\infty\gamma^tP(s_t=s|\pi)
ρ π ( s , a ) = π ( a ∣ s ) t = 0 ∑ ∞ γ t P ( s t = s ∣ π ) π \pi π D \mathcal D D π ∈ Π \pi \in \Pi π ∈ Π ρ ∈ D \rho\in\mathcal D ρ ∈ D
若ρ ∈ D \rho\in\mathcal D ρ ∈ D ρ \rho ρ π ρ = ρ ( s , a ) / ∑ a ′ ρ ( s , a ′ ) \pi_{\rho}=\rho(s,a)/\sum\limits_{a'}\rho(s,a') π ρ = ρ ( s , a ) / a ′ ∑ ρ ( s , a ′ ) π ρ \pi_{\rho} π ρ
根据Definition 1 ，可以将γ \gamma γ E π [ c ( s , a ) ] = ∑ s , a ρ π ( s , a ) c ( s , a )
\mathbb E_\pi[c(s,a)]=\sum\limits_{s,a}\rho_\pi(s,a)c(s,a)
E π [ c ( s , a ) ] = s , a ∑ ρ π ( s , a ) c ( s , a )
若H ( π ) = E π [ − log π ( a ∣ s ) ] H(\pi)=\mathbb E_\pi[-\log\pi(a|s)] H ( π ) = E π [ − log π ( a ∣ s ) ] H ‾ ( ρ ) = − ∑ s , a ρ ( s , a ) log ( ρ ( s , a ) / ∑ a ′ ρ ( s , a ′ ) ) \overline H(\rho)=-\sum\limits_{s,a}\rho(s,a)\log(\rho(s,a)/ \sum_{a'}\rho(s,a')) H ( ρ ) = − s , a ∑ ρ ( s , a ) log ( ρ ( s , a ) / ∑ a ′ ρ ( s , a ′ ) ) H ‾ \overline H H π ∈ Π , ρ ∈ D \pi \in \Pi, \rho \in \mathcal D π ∈ Π , ρ ∈ D H ( π ) = H ‾ ( ρ π ) H(\pi)=\overline H(\rho_\pi) H ( π ) = H ( ρ π ) H ( π ρ ) = H ‾ ( ρ ) H(\pi_\rho)=\overline H(\rho_) H ( π ρ ) = H ( ρ )
若L ( π , c ) = − H ( π ) + E π [ c ( s , a ) ] L(\pi,c)=-H(\pi)+\mathbb E_\pi[c(s,a)] L ( π , c ) = − H ( π ) + E π [ c ( s , a ) ] L ‾ ( ρ , c ) = − H ‾ ( ρ ) + ∑ s , a ρ ( s , a ) c ( s , a ) \overline L(\rho,c) = -\overline H(\rho) + \sum_{s,a}\rho(s,a)c(s,a) L ( ρ , c ) = − H ( ρ ) + ∑ s , a ρ ( s , a ) c ( s , a ) c c c π ∈ Π \pi\in\Pi π ∈ Π L ( π , c ) = L ‾ ( ρ π , c ) L(\pi,c)=\overline L(\rho_\pi,c) L ( π , c ) = L ( ρ π , c ) ρ ∈ D \rho\in\mathcal D ρ ∈ D L ( π ρ , c ) = L ‾ ( ρ , c ) L(\pi_\rho,c)=\overline L(\rho,c) L ( π ρ , c ) = L ( ρ , c )
对于一个方程f : R R × A → R ‾ f:\mathbb R^{\mathcal R \times \mathcal A}\to \overline \mathbb R f : R R × A → R f ∗ : R R × A → R ‾ f^*:\mathbb R^{\mathcal R \times \mathcal A} \to \overline \mathbb R f ∗ : R R × A → R f ∗ ( x ) = sup y ∈ R S × A x T y − f ( y ) f^*(x) = \sup_{y\in\mathbb R^{\mathcal S \times \mathcal A}}x^Ty-f(y) f ∗ ( x ) = sup y ∈ R S × A x T y − f ( y )
$RL(\widetilde c) $是利用IRL恢复的cost，通过RL学得的policy。可得R L ∘ I R L ψ ( π E ) = a r g m i n π ∈ Π − H ( π ) + ψ ∗ ( ρ π − ρ π E )
RL\circ IRL_\psi(\pi_E) = argmin_{\pi\in\Pi}-H(\pi) + \psi^*(\rho_\pi - \rho_{\pi_{E}})
R L ∘ I R L ψ ( π E ) = a r g m i n π ∈ Π − H ( π ) + ψ ∗ ( ρ π − ρ π E ) ψ \psi ψ ψ ∗ \psi^* ψ ∗ ( c , π ) (c,\pi) ( c , π ) ψ \psi ψ ( c , π ) (c,\pi) ( c , π )
在本文中将会主要介绍三种不同的正则化函数：恒定正则化函数，示性正则化函数，生成对抗正则化函数（GA）
若ψ \psi ψ c ~ ∈ I R L ψ ( π E ) \widetilde c\in IRL_\psi(\pi_E) c ∈ I R L ψ ( π E ) π ~ ∈ R L ( c ~ ) \widetilde \pi\in RL(\widetilde c) π ∈ R L ( c ) ρ π ~ = ρ π E \rho_{\widetilde \pi} = \rho_{\pi_E} ρ π = ρ π E
若没有正则化项，则RL得到的policy将会精确匹配专家policy
但该算法无法应用于实际系统中，因为实际系统的环境非常大，计算复杂。
学徒算法是想要找到一个policy并且比专家policy效果在学得的cost function C \mathcal C C min π max c ∈ C E π [ c ( s , a ) ] − E π E [ c ( s , a ) ]
\min_\pi \max_{c\in\mathcal C} \mathbb E_\pi[c(s,a)] - \mathbb E_{\pi_E}[c(s,a)]
π min c ∈ C max E π [ c ( s , a ) ] − E π E [ c ( s , a ) ] C \mathcal C C f 1 , f 2 , … , f d f_1,f_2,\dots,f_d f 1 , f 2 , … , f d C l i n e a r = { ∑ i w i f i : ∥ w ∥ 2 ≤ 1 } \mathcal C_{linear} = \{ \sum_i w_if_i:\|w\|_2\le1\} C l i n e a r = { ∑ i w i f i : ∥ w ∥ 2 ≤ 1 } C c o n v e x = { ∑ i w i f i : ∑ i w i = 1 , w i ≥ 0 ∀ i } \mathcal C_{convex}=\{ \sum_i w_if_i:\sum_iw_i=1,w_i\ge 0 \forall i\} C c o n v e x = { ∑ i w i f i : ∑ i w i = 1 , w i ≥ 0 ∀ i }
对于示性函数δ c : R S × A → R ‾ \delta_c:\mathbb R^{\mathcal S \times \mathcal A}\to \overline \mathbb R δ c : R S × A → R δ C ( c ) = { 0 , c ∈ C + ∞ , o t h e r w i s e
\delta_{\mathcal C}(c) =\left\{
\begin{array}{lr}
0, & c\in\mathcal C \\
+\infty, & otherwise
\end{array}
\right.
δ C ( c ) = { 0 , + ∞ , c ∈ C o t h e r w i s e max c ∈ C E π [ c ( s , a ) ] − E π E [ c ( s , a ) ] = max c ∈ C − δ C ( c ) + ∑ s , a ( ρ π ( s , a ) − ρ π E ( s , a ) ) c ( s , a ) = δ C ∗ ( ρ π − ρ π E )
\begin{aligned}
&\max_{c\in\mathcal C} \mathbb E_\pi[c(s,a)] - \mathbb E_{\pi_E}[c(s,a)] \\
&=\max_{c\in\mathcal C} -\delta_{\mathcal C}(c)+\sum_{s,a}(\rho_\pi(s,a)-\rho_{\pi_E}(s,a))c(s,a) \\
&=\delta_{\mathcal C}^*(\rho_\pi - \rho_{\pi_E})
\end{aligned}
c ∈ C max E π [ c ( s , a ) ] − E π E [ c ( s , a ) ] = c ∈ C max − δ C ( c ) + s , a ∑ ( ρ π ( s , a ) − ρ π E ( s , a ) ) c ( s , a ) = δ C ∗ ( ρ π − ρ π E )
min π − H ( π ) + max c ∈ C E π [ c ( s , a ) ] − E π E [ c ( s , a ) ] = min π − H ( π ) + δ C ∗ ( ρ π − ρ π E )
\min_\pi - H(\pi)+\max_{c\in\mathcal C}\mathbb E_\pi[c(s,a)] - \mathbb E_{\pi_E}[c(s,a)] = \min_{\pi} -H(\pi)+\delta_{\mathcal C}^*(\rho_\pi-\rho_{\pi_E})
π min − H ( π ) + c ∈ C max E π [ c ( s , a ) ] − E π E [ c ( s , a ) ] = π min − H ( π ) + δ C ∗ ( ρ π − ρ π E ) ψ = δ C \psi=\delta_{\mathcal C} ψ = δ C
ψ G A ( c ) = { E π E [ g ( c ( s , a ) ) ] , c < 0 + ∞ , o t h e r w i s e g ( x ) = { − x − log ( 1 − e x ) , x < 0 0 , o t h e r w i s e
\psi_{GA}(c) = \left\{
\begin{array}{lr}
\mathbb E_{\pi_E}[g(c(s,a))],\qquad &c<0 \\
+\infty, & otherwise
\end{array}
\right. \\
g(x)=\left\{
\begin{array}{lr}
-x-\log(1-e^x),\qquad &x<0 \\
0, & otherwise
\end{array}
\right.
ψ G A ( c ) = { E π E [ g ( c ( s , a ) ) ] , + ∞ , c < 0 o t h e r w i s e g ( x ) = { − x − log ( 1 − e x ) , 0 , x < 0 o t h e r w i s e
ψ G A \psi_{GA} ψ G A
不像δ C \delta_{\mathcal C} δ C ψ G A \psi_{GA} ψ G A
选择ψ G A \psi_{GA} ψ G A ψ G A ∗ ( ρ π − ρ π E ) = max D ∈ ( 0 , 1 ) S × A E π E [ log ( D ( s , a ) ) ] + E π [ log ( 1 − D ( s , a ) ) ]
\psi_{GA}^*(\rho_\pi-\rho_{\pi_E}) = \max_{D\in(0,1)^{\mathcal S \times \mathcal A}} \mathbb E_{\pi_E}[\log(D(s,a))] + \mathbb E_\pi[\log(1-D(s,a))]
ψ G A ∗ ( ρ π − ρ π E ) = D ∈ ( 0 , 1 ) S × A max E π E [ log ( D ( s , a ) ) ] + E π [ log ( 1 − D ( s , a ) ) ] π \pi π π E \pi_E π E D J S ( ρ π , ρ π E ) = D K L ( ρ π ∥ ( ρ π − ρ π E ) / 2 ) + D K L ( ρ π E ∥ ( ρ E + ρ π E ) / 2 )
D_{JS}(\rho_\pi,\rho_{\pi_E}) = D_{KL}(\rho_\pi \|(\rho_\pi - \rho_{\pi_E})/2)+D_{KL}(\rho_{\pi_E}\|(\rho_E+\rho_{\pi_E})/2)
D J S ( ρ π , ρ π E ) = D K L ( ρ π ∥ ( ρ π − ρ π E ) / 2 ) + D K L ( ρ π E ∥ ( ρ E + ρ π E ) / 2 ) min π ψ G A ∗ ( ρ π − ρ π E ) − λ H ( π ) ⟺ min π max D E π E [ log ( D ( s , a ) ) ] + E π [ log ( 1 − D ( s , a ) ] − λ H ( π ) ⟺ min π D J S ( ρ π , ρ π E ) − λ H ( π )
\min_{\pi}\psi_{GA}^*(\rho_\pi - \rho_{\pi_E})-\lambda H(\pi) \Longleftrightarrow \\
\min_{\pi}\max_{D}\mathbb E_{\pi_E}[\log(D(s,a))]+\mathbb E_{\pi}[\log(1-D(s,a)]-\lambda H(\pi) \Longleftrightarrow \\
\min_{\pi}D_{JS}(\rho_\pi,\rho_{\pi_E}) - \lambda H(\pi)
π min ψ G A ∗ ( ρ π − ρ π E ) − λ H ( π ) ⟺ π min D max E π E [ log ( D ( s , a ) ) ] + E π [ log ( 1 − D ( s , a ) ] − λ H ( π ) ⟺ π min D J S ( ρ π , ρ π E ) − λ H ( π )
min G max D E x ∼ P d a t a ( x ) [ log D ( x ) ] + E z ∼ P z ( z ) [ log ( 1 − D ( G ( z ) ) ) ]
\min_G\max_D \mathbb E_{x\sim P_{data}(x)}[\log D(x)] + \mathbb E_{z\sim P_z(z)}[\log(1-D(G(z)))]
G min D max E x ∼ P d a t a ( x ) [ log D ( x ) ] + E z ∼ P z ( z ) [ log ( 1 − D ( G ( z ) ) ) ]
学习到的policy的占用率度量ρ π \rho_\pi ρ π
专家经验的占用率度量ρ π E \rho_{\pi_E} ρ π E
期望找到如下式的鞍点( π , D ) (\pi,D) ( π , D ) E π E [ log ( D ( s , a ) ) ] + E π [ log ( 1 − D ( s , a ) ) ] − λ H ( π )
\mathbb E_{\pi_E}[\log(D(s,a))] + \mathbb E_\pi[\log(1-D(s,a))]-\lambda H(\pi)
E π E [ log ( D ( s , a ) ) ] + E π [ log ( 1 − D ( s , a ) ) ] − λ H ( π )
π θ \pi_\theta π θ θ \theta θ D ω D_\omega D ω ω \omega ω
对ω \omega ω
对θ \theta θ
TPRO能够保证π θ i + 1 \pi_{\theta_{i+1}} π θ i + 1 π θ i \pi_{\theta_{i}} π θ i