Sovereign 8s.is mirror of standardgalactic/TEMPO
  • Python 90.6%
  • Shell 9.3%
  • Jinja 0.1%
Find a file
Repository files (latest commit first)
Filename Latest commit message Latest commit date
Qingyang Zhang dbf313c379
Some checks failed
e2e_ascend / verl Ascend test (self-host) (push) Has been skipped
e2e_sft / setup (push) Has been skipped
e2e_sft / e2e_sft (push) Has been skipped
model / setup (push) Has been skipped
model / model_rmpad (push) Has been skipped
model / model_rmpad_fsdp2_unstable (push) Has been skipped
model / mcore_config_converter (push) Has been skipped
model / model_engine (push) Has been skipped
reward_model / setup (push) Has been skipped
reward_model / reward_model (push) Has been skipped
checkpoint_converter / setup (push) Has been skipped
checkpoint_converter / checkpoint_converter (push) Has been skipped
checkpoint_converter / checkpoint_converter_large_moe_models (push) Has been skipped
cpu_unit_tests / cpu_unit_tests (push) Has been skipped
checkpoint_converter / cleanup (push) Has been cancelled
doc_test / doc_test (3.10) (push) Has been cancelled
e2e_sft / cleanup (push) Has been cancelled
vllm / setup (push) Has been skipped
vllm / vllm (push) Has been skipped
model / cleanup (push) Has been cancelled
pre-commit / pre-commit (3.12) (push) Has been cancelled
reward_model / cleanup (push) Has been cancelled
sanity / sanity (3.10) (push) Has been cancelled
Scorecard supply-chain security / Scorecard analysis (push) Has been cancelled
/ test (push) Has been cancelled
vllm / cleanup (push) Has been cancelled
Update README.md
2026-04-14 17:33:41 +08:00
.github init 2025-10-11 00:19:15 +08:00
docker init 2025-10-11 00:19:15 +08:00
docs init 2025-10-11 00:19:15 +08:00
examples Add TEMPO changes: empo trainer, updated reward managers, modified README and configs 2026-04-14 17:32:30 +08:00
recipe Add TEMPO changes: empo trainer, updated reward managers, modified README and configs 2026-04-14 17:32:30 +08:00
scripts Add TEMPO changes: empo trainer, updated reward managers, modified README and configs 2026-04-14 17:32:30 +08:00
tests Add TEMPO changes: empo trainer, updated reward managers, modified README and configs 2026-04-14 17:32:30 +08:00
verl Add TEMPO changes: empo trainer, updated reward managers, modified README and configs 2026-04-14 17:32:30 +08:00
.gitignore init 2025-10-11 00:19:15 +08:00
.pre-commit-config.yaml init 2025-10-11 00:19:15 +08:00
.readthedocs.yaml init 2025-10-11 00:19:15 +08:00
CONTRIBUTING.md init 2025-10-11 00:19:15 +08:00
LICENSE init 2025-10-11 00:19:15 +08:00
Notice.txt init 2025-10-11 00:19:15 +08:00
pyproject.toml init 2025-10-11 00:19:15 +08:00
README.md Update README.md 2026-04-14 17:33:41 +08:00
requirements-cuda.txt init 2025-10-11 00:19:15 +08:00
requirements-npu.txt init 2025-10-11 00:19:15 +08:00
requirements.txt init 2025-10-11 00:19:15 +08:00
requirements_sglang.txt init 2025-10-11 00:19:15 +08:00
setup.py init 2025-10-11 00:19:15 +08:00

TEMPO: Scaling Test-time Training for Large Reasoning Models

TEMPO is a semi-supervised reinforcement learning framework for scaling test-time training (TTT) of large reasoning models. It alternates between a Critic Recalibration step (E-step) on labeled data and a Policy Refinement step (M-step) on unlabeled data, enabling effective use of both labeled and unlabeled problem sets.


News

  • [2026-04] TEMPO code released.

Introduction

Test-time training (TTT) with reinforcement learning (RL) improves LLM reasoning by rolling out and learning from model-generated solutions. However, standard online RL (e.g., GRPO, DAPO) requires reward labels for every training prompt, which limits scalability when labeled data is scarce.

TEMPO frames TTT as an Expectation-Maximization (EM) problem:

  • E-step (Critic Recalibration): A value critic is updated on a small labeled dataset D_L to estimate solution quality.
  • M-step (Policy Refinement): The policy is updated on a larger unlabeled dataset D_U using critic-assigned advantages, without needing ground-truth rewards.

This allows TEMPO to leverage far more training prompts than labeled-only methods while maintaining the exploration diversity that purely outcome-supervised methods sacrifice.


Key Results

Main Benchmark Results (pass@1, greedy)

Model Benchmark Base TEMPO Gain
OLMo3-7B AIME 2024 33.0 51.1 +18.1
OLMo3-7B AIME 2025 26.3 37.0 +10.7
OLMo3-7B BeyondAIME 17.6 24.5 +6.9
Qwen3-8B AIME 2024 26.3 42.7 +16.4
Qwen3-8B AIME 2025 25.4 40.8 +15.4
Qwen3-14B AIME 2024 42.3 65.8 +23.5
Qwen3-14B AIME 2025 37.1 44.6 +7.5

Results for OLMo3-7B reported at 256 training steps; Qwen3-14B at 224 steps (training continues to improve beyond these checkpoints).

Diversity Preservation

Unlike TTRL and EMPO, which improve mean accuracy at the cost of solution diversity (pass@k degrades), TEMPO preserves and improves both mean accuracy and pass@k. This reflects a fundamentally different learning dynamic: TEMPO does not collapse the model's exploration capability.


Installation

TEMPO is built on top of verl. Install the base dependencies first:

git clone https://github.com/QingyangZhang/TEMPO.git
cd TEMPO
pip install -e .
pip install -r requirements-cuda.txt

Additional requirements for vLLM rollout:

pip install vllm>=0.8.0

Quick Start

Training scripts are located in examples/ppo_trainer/tempo/.

OLMo3-7B on AIME

export MODEL_PATH=/path/to/actor_model
export CRITIC_PATH=/path/to/critic_model
export CKPTS_DIR=./checkpoints/tempo-olmo3-7b
export train_files=/path/to/train_data.parquet
export TEST_DATA_DIR=/path/to/test_data

bash examples/ppo_trainer/tempo/run_em_olmo3_7b_aime.sh

Qwen3-14B on AIME

export MODEL_PATH=/path/to/actor_model
export CRITIC_PATH=/path/to/critic_model
export CKPTS_DIR=./checkpoints/tempo-qwen3-14b
export train_files=/path/to/train_data.parquet
export TEST_DATA_DIR=/path/to/test_data

bash examples/ppo_trainer/tempo/run_em_qwen3_14b_aime.sh

Evaluation

# Evaluate pass@k
bash examples/ppo_trainer/tempo/eval_pass_k.sh

# Evaluate pass@1 (greedy)
bash examples/ppo_trainer/tempo/eval.sh

Key Training Arguments

Argument Description
algorithm.adv_estimator=em_token Use token-level EM advantage estimator
+algorithm.filter_groups.enable=True Enable semi-supervised group filtering
critic.self_critic=True Enable critic recalibration (E-step)
critic.enable=True Enable critic model
+trainer.iter_ttrl=true Enable iterative TTT loop
data.train_files Labeled training data (parquet)
data.val_files Validation data (list of parquets)

Algorithm Overview

for each training step:
    # E-step: Critic Recalibration
    collect labeled rollouts from D_L
    update critic on labeled data with reward supervision

    # M-step: Policy Refinement
    collect unlabeled rollouts from D_U
    compute advantages using recalibrated critic
    update policy via GSPO/PPO loss

The labeled dataset D_L provides reward signal to recalibrate the critic; the unlabeled dataset D_U provides a much larger and more diverse set of training prompts for the policy. The critic bridges the two, acting as a soft reward model over unlabeled data.


Acknowledgements

TEMPO is built on top of verl by ByteDance. We thank the verl team for their open-source RL training infrastructure.

This work was conducted at Tianjin University, Tongyi Lab, and Shanghai AI Lab.