- Python 100%
| Filename | Latest commit message | Latest commit date |
|---|---|---|
| data | ||
| gapo | ||
| patches | ||
| scripts | ||
| .gitattributes | ||
| .gitignore | ||
| pyproject.toml | ||
| README.md | ||
| requirements.txt | ||
| test.sh | ||
GAPO: Group Adaptive Clipping Policy Optimization
What is here is a self-contained plug-in version of the method, enough to reproduce most of our experiments. The original code from this project that produced every detailed setup is going through Amazon's release review, and I will add it here as that clears.
Most of this work was done back in May 2026 on small setups, with 1.5B models, short context and a modest compute budget. We hope to add showcases from larger-scale RL and agentic RL training soon.
Questions are welcome at sheng.jia@mail.utoronto.ca or shengama@amazon.com.
Method
Group-relative policy optimization for RLVR normally uses a fixed importance-sampling ratio clipping boundary across all rollouts, so rare correct rollouts on harder problems are suppressed at the same rate as abundant correct rollouts on easier ones, despite carrying a stronger gradient signal. GAPO adapts the clipping boundary to the rollout advantage instead, requiring no reward shaping and preserving the standard PPO/GSPO surrogate while changing only the clipping threshold.
Setup
Our runs used 8xH200 with Python 3.11, torch 2.9.0+cu128, vLLM 0.12.0, flash-attn 2.8.3 and ray 2.55.0.
export WORK_DIR=$HOME/gapo-workspace
mkdir -p "$WORK_DIR" && cd "$WORK_DIR"
python3.11 -m venv .venv && source .venv/bin/activate
pip install torch==2.9.0 --index-url https://download.pytorch.org/whl/cu128
pip install vllm==0.12.0
pip install flash-attn==2.8.3 --no-build-isolation
git clone https://github.com/verl-project/verl.git
git -C verl checkout 9bda8b9a
pip install -e ./verl
git clone https://github.com/Sheng-J/GAPO.git && cd GAPO
pip install -e . && pip install -r requirements.txt
python data/prepare_data.py --out-dir "$WORK_DIR/data"
git -C "$WORK_DIR/verl" apply "$WORK_DIR/GAPO/patches/gapo-verl-9bda8b9a.patch"
The patch wires the adaptive clip into verl and makes it usable together with verl's
bypass rollout correction, which the runs rely on for off-policy correction. In this
version of verl, enabling bypass forces the actor onto verl's own bypass loss, which
supports neither the GSPO sequence ratio nor a per-rollout clip, so bypass and GAPO cannot
otherwise be combined. Pin verl to 9bda8b9a, since later revisions move some of the call
sites the patch touches. The scripts will not work without it.
Running
export WORK_DIR=$HOME/gapo-workspace
bash scripts/train_gapo_qwen25_math_1.5b_eps7e4_15e4_n16.sh
test.sh wraps this for convenience, launching in the background and printing how to
follow or stop the run: bash test.sh eps7e4 15 runs the 7e-4/1.5e-3 configuration for 15
epochs.
| script | model | response length | eps_lo -> eps_hi_max |
|---|---|---|---|
..._qwen25_math_1.5b_eps7e4_15e4_n16.sh |
Qwen2.5-Math-1.5B | 3072 | 7e-4 -> 1.5e-3 |
..._qwen25_math_1.5b_eps3e3_5e3_n16.sh |
Qwen2.5-Math-1.5B | 3072 | 3e-3 -> 5e-3 |
..._r1distill_1.5b_24k_eps7e5_3e4_n16.sh |
R1-Distill-Qwen-1.5B | 24576 | 7e-5 -> 3.0e-4 |
These use 16 rollouts per prompt, which we found more stable. Each has an _n8 counterpart
with 8, for a smaller compute budget.
Metrics go to the console. To also log to Weights and Biases, export a key before launching and the scripts will pick it up:
export WANDB_API_KEY=... # from wandb.ai/authorize
export WANDB_PROJECT=gapo # optional
Acknowledgements
GAPO is built as a plug-in for verl, which provides the training infrastructure these runs rely on. Thanks to the verl authors and maintainers.
@article{sheng2024hybridflow,
title = {HybridFlow: A Flexible and Efficient RLHF Framework},
author = {Guangming Sheng and Chi Zhang and Zilingfeng Ye and Xibin Wu and Wang Zhang and Ru Zhang and Yanghua Peng and Haibin Lin and Chuan Wu},
year = {2024},
journal = {arXiv preprint arXiv: 2409.19256}
}