Sovereign 8s.is mirror of standardgalactic/GAPO
Find a file
Repository files (latest commit first)
Filename Latest commit message Latest commit date
2026-09-02 04:34:15 +00:00
data Initial release of GAPO plug-in implementation for EMNLP 2026-08-30 14:57:34 +00:00
gapo Initial release of GAPO plug-in implementation for EMNLP 2026-08-30 23:52:28 +00:00
patches Aggregate the GAPO loss with verl's agg_loss instead of over the local micro-batch 2026-08-30 23:20:24 +00:00
scripts push more launch scripts 2026-09-02 04:17:48 +00:00
.gitattributes Initial release of GAPO plug-in implementation for EMNLP 2026-08-30 23:52:28 +00:00
.gitignore Initial release of GAPO plug-in implementation for EMNLP 2026-08-30 14:57:34 +00:00
pyproject.toml Initial release of GAPO plug-in implementation for EMNLP 2026-08-30 14:57:34 +00:00
README.md Use absolute path for the verl patch 2026-09-02 04:34:15 +00:00
requirements.txt Initial release of GAPO plug-in implementation for EMNLP 2026-08-30 14:57:34 +00:00
test.sh push more launch scripts 2026-09-02 04:17:48 +00:00

GAPO: Group Adaptive Clipping Policy Optimization

What is here is a self-contained plug-in version of the method, enough to reproduce most of our experiments. The original code from this project that produced every detailed setup is going through Amazon's release review, and I will add it here as that clears.

Most of this work was done back in May 2026 on small setups, with 1.5B models, short context and a modest compute budget. We hope to add showcases from larger-scale RL and agentic RL training soon.

Questions are welcome at sheng.jia@mail.utoronto.ca or shengama@amazon.com.

Method

Group-relative policy optimization for RLVR normally uses a fixed importance-sampling ratio clipping boundary across all rollouts, so rare correct rollouts on harder problems are suppressed at the same rate as abundant correct rollouts on easier ones, despite carrying a stronger gradient signal. GAPO adapts the clipping boundary to the rollout advantage instead, requiring no reward shaping and preserving the standard PPO/GSPO surrogate while changing only the clipping threshold.

Setup

Our runs used 8xH200 with Python 3.11, torch 2.9.0+cu128, vLLM 0.12.0, flash-attn 2.8.3 and ray 2.55.0.

export WORK_DIR=$HOME/gapo-workspace
mkdir -p "$WORK_DIR" && cd "$WORK_DIR"

python3.11 -m venv .venv && source .venv/bin/activate
pip install torch==2.9.0 --index-url https://download.pytorch.org/whl/cu128
pip install vllm==0.12.0
pip install flash-attn==2.8.3 --no-build-isolation

git clone https://github.com/verl-project/verl.git
git -C verl checkout 9bda8b9a
pip install -e ./verl

git clone https://github.com/Sheng-J/GAPO.git && cd GAPO
pip install -e . && pip install -r requirements.txt
python data/prepare_data.py --out-dir "$WORK_DIR/data"
git -C "$WORK_DIR/verl" apply "$WORK_DIR/GAPO/patches/gapo-verl-9bda8b9a.patch"

The patch wires the adaptive clip into verl and makes it usable together with verl's bypass rollout correction, which the runs rely on for off-policy correction. In this version of verl, enabling bypass forces the actor onto verl's own bypass loss, which supports neither the GSPO sequence ratio nor a per-rollout clip, so bypass and GAPO cannot otherwise be combined. Pin verl to 9bda8b9a, since later revisions move some of the call sites the patch touches. The scripts will not work without it.

Running

export WORK_DIR=$HOME/gapo-workspace
bash scripts/train_gapo_qwen25_math_1.5b_eps7e4_15e4_n16.sh

test.sh wraps this for convenience, launching in the background and printing how to follow or stop the run: bash test.sh eps7e4 15 runs the 7e-4/1.5e-3 configuration for 15 epochs.

script model response length eps_lo -> eps_hi_max
..._qwen25_math_1.5b_eps7e4_15e4_n16.sh Qwen2.5-Math-1.5B 3072 7e-4 -> 1.5e-3
..._qwen25_math_1.5b_eps3e3_5e3_n16.sh Qwen2.5-Math-1.5B 3072 3e-3 -> 5e-3
..._r1distill_1.5b_24k_eps7e5_3e4_n16.sh R1-Distill-Qwen-1.5B 24576 7e-5 -> 3.0e-4

These use 16 rollouts per prompt, which we found more stable. Each has an _n8 counterpart with 8, for a smaller compute budget.

Metrics go to the console. To also log to Weights and Biases, export a key before launching and the scripts will pick it up:

export WANDB_API_KEY=...          # from wandb.ai/authorize
export WANDB_PROJECT=gapo         # optional

Acknowledgements

GAPO is built as a plug-in for verl, which provides the training infrastructure these runs rely on. Thanks to the verl authors and maintainers.

@article{sheng2024hybridflow,
  title   = {HybridFlow: A Flexible and Efficient RLHF Framework},
  author  = {Guangming Sheng and Chi Zhang and Zilingfeng Ye and Xibin Wu and Wang Zhang and Ru Zhang and Yanghua Peng and Haibin Lin and Chuan Wu},
  year    = {2024},
  journal = {arXiv preprint arXiv: 2409.19256}
}