Open Reasoner Zero

An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model

Paper PDF Link [WIP]👁️

Figure 1 | Evaluation performance of Open-Reasoner-Zero-{7B, 32B}. We report the average accuracy on the benchmark dataset for each question with 16 responses. Notably, Open-Reasoner-Zero-32B outperforms DeepSeek-R1-Zero-Qwen-32B on the GPQA Diamond benchmark while only requiring 1/30 of the training steps. We are continuing to scale up these RL settings until this preprint is released, as there is no sign of saturation.

Figure 2 | Train Time Scale up both on Reward and Response Length of Open-Reasoner-Zero-{7B, 32B}.

Overview

🌊 We introduce Open-Reasoner-Zero, the first open source implementation of large-scale reasoning-oriented RL training focusing on scalability, simplicity and accessibility.

To enable broader participation in this pivotal moment we witnessed and accelerate research towards artificial general intelligence (AGI), we release our source code, parameter settings, training data, and model weights. Please refer to our paper for more insights.

Let the Reasoner-Zero tide rise!

Releases 📦

[2025/02/18] We release Open-Reasoner-Zero.

As part of this release, we open-source:

• 🌊 Paper on our comprehensive analysis and insights in Reasoner-Zero training
• 🤗 HF Model Open-Reasoner-Zero-7B and Open-Reasoner-Zero-32B
• 🎁 Our curated 57k training data
• 📄 Training Scripts to enjoy your own Reasoner-Zero journey!

Key Features in Codebase 🔑

• Adopt single controller trainer design, flexible and researcher-friendly.
• Colocate training and generation in the same GPUs to maximize GPU utilization.

Getting Started 🚀

Installation & Training Scripts

We release our Dockerfile in docker folder to facilitate the reproducibility of our training.

To install the package, run:

pip install -e .

Start Orz-7B PPO Training

debug running command:

# NOTE: just for debug, not final setting!

## Debug command in a single GPU with `EleutherAI/pythia-14m`
DEBUG_MODE=True python -m playground.orz_14m_ppo_mini

## Debug command in a single node (8 GPUs) with `Qwen/Qwen2.5-7B`
DEBUG_MODE=True python -m playground.orz_7b_ppo

Multi-node Training on 4 nodes:

first on master node, run:

ray start --head

then on other nodes, run:

ray start --address='<master-node-ip>:<master-node-port>'

then on master node, run:

python -m playground.orz_7b_ppo

Your training log will be shown in the master node terminal.

Start Orz-32B PPO Training

running command in 16 nodes:

first on master node, run:

ray start --head

then on other nodes, run:

ray start --address='<master-node-ip>:<master-node-port>'

then on master node, run:

python -m playground.orz_32b_ppo

Your training log will be shown in the master node terminal.

Data

We release all of 57k curated high-quality training data in the data folder.

The details for how to collect data are described in our paper.

Acknowledgements

• This work was supported by computing resources and valuable feedback provided by StepFun and Tsinghua University.
• Our training framework is built on OpenRLHF, vllm, DeepSpeed and ray.
• Our model is based on Qwen2.5-7B and Qwen2.5-32B.
• We thank Project Numina and Tulu3 for their collected open sourced data.

Advertisement Time 📣

We are hiring talented researchers and engineers to join our team. If you are interested in our project and would like to contribute to the reasoner scale-up all the way to AGI, please feel free to reach out to us at [email protected]

Community Discussions

We have a wechat group to help discussions and sharing, you can scan the QR code below to join.

Citation

@misc{OpenReasonerZero2025,
  title={Open-Reasoner-Zero: An Open Source Approach to Scaling Reinforcement Learning on the Base Model},
  author={Jingcheng Hu and Yinmin Zhang and Qi Han and Daxin Jiang and Xiangyu Zhang, Heung-Yeung Shum},
  year={2025},
  howpublished={\url{https://github.com/Open-Reasoner-Zero/Open-Reasoner-Zero}},
}