Guangming Sheng (@PeterSheng66) header
Guangming Sheng (@PeterSheng66) profile picture

PhD Student @HKUnversity | lead contributor of veRL @verl_project (HybridFlow) 👽Code mirrors mind. Karma stays yours

Joined July 2019
Excited to announce veRL: a Flexible, Efficient RL framework for LLM Post-Training 🚀 Code: github.com/volcengine/verl Paper: arxiv.org/pdf/2409.19256 veRL integrate various SOTA LLM infrastrures and enables implementing RL algorithm for LLMs in a single process. #veRL #RL #LLM
1
2
577
Genius!
Welcome to enjoy ICLR Expo Talk Panel "verl: Flexible and Efficient Infrastructures for Post-training LLMs" from ByteDance Seed! - Link: iclr.cc/Expo/Conferences/202… - Time: Sat 26 Apr 1 (Today!) p.m. - 2 p.m. +08 - Location: Peridot 202-203 (2nd Floor in Expo) - Contents: - "verl: Flexible and Efficient RL for LLMs" - "DAPO & VAPO: An Open-Source LLM RL System at Scale" - Related Resources: github.com/eric-haibin-lin/v… (or scan the QR code) If we compare the development of AI to the cultivation of a noble person (Junzi, 君子), then Reinforcement Learning (RL) would be analogous to the Confucian concept of "investigating things to extend knowledge" (格物致知). Ancient Chinese philosophy emphasizes the harmony between Yin (阴, like earth) and Yang (阳, like heaven). In RL, infrastructure is like Yin/earth, while algorithm is like Yang/heaven. The verl framework, based on the HybridFlow paradigm, combines flexibility and efficiency. Together with algorithms like DAPO and VAPO, it is push RL of AI towards the Confucian ideal of "acting freely without overstepping boundaries" (从心所欲不逾矩).
2
313
Guangming Sheng retweeted
🚀 Excited to share DAPO! The current state-of-the-art large-scale LLM reinforcement learning system, which is also open-sourced! 🔥RL performance Based on the Qwen-2.5-32B pretrained model, DAPO gets 50 points on AIME. This achieves the new SOTA performance using 50% training steps (previous SOTA achieved by DeepSeek R1's GRPO, 47 points on AIME) 🌟Fully open-sourced! We release all algorithm details, training code, and datasets. Project page: dapo-sia.github.io/ Paper: arxiv.org/pdf/2503.14476 Code & Dataset: github.com/BytedTsinghua-SIA… Technical details introduced below #GRPO #DOUBAO #bytedance #LLM #RL #DAPO #Deepseek #Verl
13
44
9
215
28,206
One more large scale deployment of verl project!
Replying to @qiying_yu
@qiying_yu and team just dropped the DAPO algorithm (decoupled clip and dynamic sampling policy optimization)! DAPO-Zero-32B, a fully open-source RL reasoning model, surpasses DeepSeek-R1-Zero-Qwen-32B, and scores 50 on AIME 2024 with 50% fewer steps. It is trained with **zero-shot RL** from the Qwen-32b pre-trained model. Everything is fully open-sourced (algorithm, code, dataset, verifier, and model), built with @verl_project. Homepage: dapo-sia.github.io/ Paper: dapo-sia.github.io/static/pd… Code: github.com/volcengine/verl/t…
2
193
Guangming Sheng retweeted
Replying to @qiying_yu
@qiying_yu and team just dropped the DAPO algorithm (decoupled clip and dynamic sampling policy optimization)! DAPO-Zero-32B, a fully open-source RL reasoning model, surpasses DeepSeek-R1-Zero-Qwen-32B, and scores 50 on AIME 2024 with 50% fewer steps. It is trained with **zero-shot RL** from the Qwen-32b pre-trained model. Everything is fully open-sourced (algorithm, code, dataset, verifier, and model), built with @verl_project. Homepage: dapo-sia.github.io/ Paper: dapo-sia.github.io/static/pd… Code: github.com/volcengine/verl/t…
8
112
15
450
152,374
Guangming Sheng retweeted
🙌 Huge shoutout to the dream team who made this done so brrrrrr @ManlingLi_ @James_KKW @qineng_wang @WilliamZhangNU ✨Also, to who made our proj even possible -- 🙏We thank @deepseek_ai for the based DSR1 insights; @veRL team for the robust infra; TinyZero team @jiayi_pirate Junjie Zhang @xingyaow_ @lifan__yuan for their innovative discoveries; Prof. @2prime_PKU, @pigjunebaba, and Prof. @kchonyc for their valuable guidance and discussions. Crashing now after another 40+ hrs 😴 (still better than expected!) 👀→ nitter.gq/wzihanw/status/1883541… To all celebrating – Happy Chinese New Year! 🐉🧧 May the Year of 2025 bring us closer to AGI breakthroughs that uplift humanity.
🚀 Introducing RAGEN—the world’s first reproduction of DeepSeek-R1(-Zero) methods for training agentic AI models! We’re betting big on the future of RL + LLM + Agents 🤖✨. This release is a minimally viable leap toward that vision. Code and more intro 🔗: github.com/ZihanWang314/rage…
2
2
36
10,534
Guangming Sheng retweeted
Congratulations on the Amazing Doubao-1.5 models! Doubao models are fine-tuned on veRL framework and SGLang will collaborate with veRL team deeply in the next version for a much better hybrid engine! Stay tuned!
2
11
4
69
13,245
Some interesting key challenges ByteDance ran into with its RLHF platform. - RL with 405B parameter models breaks many frameworks - During RLHF, the same LLM is used for training as well as generation in the environment simulation step. The training and generation pieces often use different engines (e.g., Megatron for training and vLLM for inference). As a consequence, weights need to be synchronized between these frameworks, which can be highly expensive. - Reconciling single-controller (e.g., Ray) and multi-controller (e.g., NCCL) programming models. It's possible to get the best of both. They spoke in more depth about the ByteDance RLHF platform here youtube.com/watch?v=MrhMcXkX…
11
30
4,559
Kudos to @raydistributed , @vllm_project , Megatron-LM @NVIDIAAI , and @PyTorch for their incredible LLM infrastructures.
78
Besides RLHF, we also provide examples of LLM reasoning tasks, such as Math & Code. Try veRL now and contribute with us! Credit to: Mentor Chi Zhang, Leader @eric_haibin_lin and Yanghua Peng, and many other colleagues in ByteDance Seed MLSys team.
1
127
Guangming Sheng retweeted
Introducing veRL by Bytedance! Flexible & efficient framework for LLM RL/RLHF. Ray-based hybrid programming model. 1.5x - 20x throughput boost. Kudos to @PeterSheng66 and team, & @raydistributed Code: github.com/volcengine/verl Paper: arxiv.org/pdf/2409.19256 #veRL #RL #LLMs
4
14
7
43
11,744