RLHF

Reinforcement Learning from Human Feedback (RLHF)

PPO Objective (Clipped Surrogate)

$L^{CLIP}(\theta )={\hat{\mathbb{E}}}_t\left[\min (r_t(\theta ){\hat{A}}_t,\text{clip}(r_t(\theta ),1-ϵ,1+ϵ){\hat{A}}_t)\right]$ where $r_t(\theta )=\frac{{\pi }_{\theta }(a_t|s_t)}{{\pi }_{\text{old}}(a_t|s_t)}$

Reward Model Objective

$L(r_{\varphi })=-{\mathbb{E}}_{(x,y_w,y_l)\sim D}\left[\log \sigma (r_{\varphi }(x,y_w)-r_{\varphi }(x,y_l))\right]$