RLVR

Reinforcement Learning with Verifiable Rewards (RLVR)

Group Relative Policy Optimization (GRPO) Objective

$L(\theta )=\mathbb{E}\left[\frac{1}{G}{\sum }_{i=1}^G\left(\min \left(\frac{{\pi }_{\theta }(a_i|s)}{{\pi }_{\text{old}}(a_i|s)}{A}_i,\text{clip}\left(\frac{{\pi }_{\theta }(a_i|s)}{{\pi }_{\text{old}}(a_i|s)},1-ϵ,1+ϵ\right)A_i\right)-\beta D_{KL}({\pi }_{\theta }\Vert {\pi }_{\text{ref}})\right)\right]$

where advantages are group-normalized: $A_i=\frac{R_i-\text{mean}(\{R_1,\dots ,R_G\})}{\text{std}(\{R_1,\dots ,R_G\})}$