Ph.D. student@Peking University
Joined March 2020
- Tweets25
- Following184
- Followers177
- Likes157
Quiet ambush๐ฆ
MiMo-V2-Pro & Omni & TTS is out. Our first full-stack model family built truly for the Agent era.
I call this a quiet ambush โ not because we planned it, but because the shift from Chat to Agent paradigm happened so fast, even we barely believed it. Somewhere in between was a process that was thrilling, painful, and fascinating all at once.
The 1T base model started training months ago. The original goal was long-context reasoning efficiency. Hybrid Attention carries real innovation, without overreaching โ and it turns out to be exactly the right foundation for the Agent era. 1M context window. MTP inference for ultra-low latency and cost. These architectural decisions weren't trendy. They were a structural advantage we built before we needed it.
What changed everything was experiencing a complex agentic scaffold โ what I'd call orchestrated Context โ for the first time. I was shocked on day one. I tried to convince the team to use it. That didn't work. So I gave a hard mandate: anyone on MiMo Team with fewer than 100 conversations tomorrow can quit. It worked. Once the team's imagination was ignited by what agentic systems could do, that imagination converted directly into research velocity.
People ask why we move so fast. I saw it firsthand building DeepSeek R1. My honest summary:
โ Backbone and Infra research has long cycles. You need strategic conviction a year before it pays off.
โ Posttrain agility is a different muscle: product intuition driving evaluation, iteration cycles compressed, paradigm shifts caught early.
โ And the constant: curiosity, sharp technical instinct, decisive execution, full commitment โ and something that's easy to underestimate: a genuine love for the world you're building for.
We will open-source โ when the models are stable enough to deserve it.
From Beijing, very late, not quite awake.
โก๏ธFrontier intelligence. Blazing speed. Unbeatable price!
MiMo-V2-Flash scores 66 on the @ArtificialAnlys Intelligence Index โ #2 among open-source models and #8 overall! ๐๐๐
Designed for Agentic AI โ now with the benchmarks to prove it: #1 on ฯยฒ-Bench Telecom for agentic tool-use among all evaluated models. โกโกโก
Frontier intelligence. Blazing speed. Unbeatable price.
Try it out ๐
๐ค Model: hf.co/XiaomiMiMo/MiMo-V2-Flaโฆ
๐จ AI Studio: aistudio.xiaomimimo.com
๐ API: platform.xiaomimimo.com
There's a new version of this post
Faster than fast. Stronger than strong.
โก Faster than Fast. Designed for Agentic AI.
Introducing Xiaomi MiMo-V2-Flash โ our new open-source MoE model: 309B total params, 15B active.
Blazing speed meets frontier performance.
๐ฅ Highlights:
๐๏ธ Hybrid Attention: 5:1 interleaved 128-window SWA + Global | 256K context
๐ Performance:
โ๏ธ Matches DeepSeek-V3.2 on general benchmarks โ at a fraction of the latency
๐ SWE-Bench Verified: 73.4% | SWE-Bench Multilingual: 71.7% โ new SOTA for open-source models
๐ Speed: 150 output tokens/s with Day-0 support from @lmsysorg๐ค
๐ค Model: hf.co/XiaomiMiMo/MiMo-V2-Flaโฆ
๐ Blog Post: mimo.xiaomi.com/blog/mimo-v2โฆ
๐ Technical Report: github.com/XiaomiMiMo/MiMo-Vโฆ
๐จ AI Studio: aistudio.xiaomimimo.com
Fighting for AGI together!
Intelligence will inevitably evolve from language to the physical world, unlocking spatial intelligence for multi-modal perception, reasoning, generation, and actionโessential for true AGI.
I'm working on building this at @XiaomiMiMo, spearheading a creative and talented team!
Yifan Song retweeted
MiMo-VL technical report, models, and evaluation suite are out!
๐ค Models: huggingface.co/XiaomiMiMo/Miโฆ (or RL)
Report: arxiv.org/abs/2506.03569
Evaluation Suite: github.com/XiaomiMiMo/lmms-eโฆ
Looking back, it's incredible that we delivered such compact yet powerful vision-language models in under six months.
Here are my key takeaways from our journey:
Reasoning is now essential for VLMs. Adding long chain-of-thought data to our training produced clear performance gains across all benchmarks. What's fascinating is watching our model actually examine different parts of images, checking various details before working through its reasoning to reach an answer.
Mixed reward learning was our biggest challenge and most inspiring discovery. We saw comprehensive improvements on almost every task with objective rewards like document perception, visual grounding, and multimodal math. MiMo-VL-RL is now the best open-source VLM on the InfoVQA test set. But subjective rewards like human preference data proved much trickierโmodels learn to game these signals surprisingly quickly. Finding the right balance is truly an art.
We're committed to reproducible VLM research. Throughout development, we experienced firsthand how difficult it is to reproduce results from other papers. Different prompts, temperature settings, and evaluation processes make fair comparisons nearly impossible. That's why we're releasing our complete evaluation suite covering 50+ tasks, built on lmms-eval, with fully reproducible results. We might be the first to do this comprehensively, and we hope it helps advance the field by making research more transparent and comparable.
Yifan Song retweeted
Today, Xiaomi releases MiMo, our first open-source reasoning model. At 7B parameters, itโs optimized for reasoning through pre-training and post-training, surpassing OpenAIโs o1-mini and QwQ-32B-Preview on AIME 2024-2025 and LiveCodeBench v5 benchmarks.
#XiaomiMiMo #MiMo7B
Yifan Song retweeted
๐ข Introducing VL-RewardBench - A new benchmark for vision-language generative reward models (VL-GenRMs)!
๐Even SOTA models struggle: Gemini-1.5-Pro (67.2%) & GPT-4o (65.8%).
๐งThe best open-source model LLaMA-3.2-90B only hits 56.2%, most others below random chance!
๐Project page: vl-rewardbench.github.io
๐Introducing MixEval-X, the first real-world, any-to-any benchmark. mixeval-x.github.io/
It extends all benefits of MixEval to multi-modal evaluations, including real-world query distribution, fast yet accurate model ranking, high standards evaluation across modalities!
๐ Frontier players are racing to solve modality puzzles in the quest for AGI.
But to get there, we need consistent, high-standard evaluations across all modalities!
๐ Introducing MixEval-X, the first real-world, any-to-any benchmark.
Inheriting the philosophy from MixEval, MixEval-X optimizes the benchmark data mixtures to reflect real-world task distributions, across diverse input and output modalities.
Several key features setting it apart:
๐ฅ Real-world task distribution, ensuring the evaluation results generalizable to real-world use cases;
๐ฅ Accurate model ranking (up to 0.98 correlation with Arena-like multi-modal evals);
๐ฅ Consistent, high standards across modalities, preventing any from lagging behind;
๐ฅ Decoupled evaluations for different modalities, compatible with models having any numbers of input / output modalities;
๐ฅ Fast, cost-effective, reproducible execution;
๐ฅ Dynamic data refresh;
๐ฅ Challenging tasks;
๐ฅ Organization-level benchmarking with balanced dimensions (modalities)โa new level of evaluation.
Project Page: mixeval-x.github.io
Paper: arxiv.org/abs/2410.13754
Github Repo: github.com/Psycoy/MixEval-X/
Data: huggingface.co/datasets/MixEโฆ
ALT The scores of frontier organizationsโ flagship models on MixEval-X.
Yifan Song retweeted
Working on multimodal instruction tuning and finding it hard to scale? Building Web/GUI agents but data is too narrow?
Introducing ๐MultiUI: 7.3M multimodal instructions from 1M webpage UIs, offering diverse data to boost text-rich visual understanding.
Key takeaways:
๐WebUI-trained models show major gains in visual web understanding and agent tasks. ๐ป
๐Models also generalize well to non-UI tasks like DocVQA/OCR. ๐
How it works:
We generate multimodal instructions with a text LLM using structured text from webpage accessibility trees. We then pair them with UI screenshots, to train multimodal models.
Homepage: neulab.github.io/MultiUI/
Paper: arxiv.org/pdf/2410.13824
Dataset: huggingface.co/datasets/neulโฆ
Model: huggingface.co/neulab/UIX-Qwโฆ
Congrats to the student lead @jeepliu1212 and the team @tianyue_01 @99Solaris @QuYuxiao @XiongChenyan @WenhuChen @gneubig !
More details are in the following threads โฌ๏ธ
Yifan Song retweeted
How to get โ๏ธChatbot Arenaโ๏ธ model rankings with 2000ร less time (5 minutes) and 5000ร less cost ($0.6)?
Maybe simply mix the classic benchmarks.
๐ Introducing MixEval, a new ๐ฅgold-standard๐ฅ LLM evaluation paradigm standing on the shoulder of giants (classic benchmarks).
mixeval.github.io/
๐ถ๏ธLLM Benchmark Mixture:
We mine comprehensive and well-distributed ๐real-world user queries from the web and match them with similar queries from off-the-shelf ๐ฏground-truth-based benchmarks.
๐คWhy to Use MixEval?
(1) ๐ฏ Accurate model ranking (0.96 correlation with Chatbot Arena)
(2) โก๏ธ Fast, cheap, and reproducible execution, requiring only 6% the time and cost of MMLU
(3) ๐ Dynamic benchmarking enabled by low-effort and stable updating mechanism
(4) ๐๏ธ Challenging question set (GPT-4o, the top model on MixEval leaderboard, achieves 64.7% accuracy)
(5) ๐ Comprehensive and highly impartial query distribution, as it is deeply grounded in real-world user queries
(6) โ๏ธ Fair grading process without preference bias, ensured by its ground-truth-based nature
โ What's wrong with the current LLM evaluation?
(1)โQuery Bias: evaluation queries falling short of
comprehensiveness or appropriate distribution
a) ground-truth-based benchmarks
b) LLM-judged benchmarks
(2)๐จโโ๏ธGrading Bias: the grading process involving
significant bias or error
a) LLM-judged benchmarks
(3)๐ฌGeneralization Bias: models overfitting the
evaluation data (contamination)
a) ground-truth-based benchmark
b) LLM-judged benchmarks
๐ค Any current benchmarks that are not so biased?
โ๏ธ Yes. Large-scale user-facing benchmarks, e.g., โ๏ธChatbot Arenaโ๏ธ, solve
(1) query bias by collecting a large number of real-world user queries,
(2) grading bias by collecting a large number of real-world user preferences, and
(3) generalization bias by continuously receiving fresh queries.
But they are prohibitively ๐ฐexpensive (around $2936 for a single model, see the below figure), โslow, and ๐ซirreproducible!
โ
MixEval addresses all these issues.
It's not only highly unbiased in query, grading, and generalization, but also fast, cheap, and reproducible.
๐We provide extensive meta-evaluation and insights for MixEval and existing LLM benchmarks in our paper. ๐ฅWe hope this will deepen the communityโs understanding of LLM evaluation and guide future research directions!
๐Our dynamic leaderboard is now available at: mixeval.github.io/#leaderboaโฆ
๐Join us in revolutionizing LLM evaluation! Test your model on MixEval and see where you stand on our dynamic leaderboard.
๐ We will update the data points on a monthly basis.
๐ Moving forward, we'll continuously add new benchmarks to our pool as they release. This will refine our mixtures and enhance dynamism at a higher level.
This work is done by @NiJinjie, @XueFz, @xiangyue96, @yuntiandeng, Mahir Shah, Kabir Jain, @gneubig, and @YangYou1991. Kudos to the team!
We also sincerely thank @Francis_YAO_ @gblazex @zhansheng @_jasonwei @p_nawrot @soldni @guanzhi_wang @deepanwayx @BoLi68567011 @JunhaoZHANG19 @99Solaris @ZangweiZheng @zian_andy_zheng @KevinQHLin @WenhuChen @billyuchenlin and colleagues from NUS HPC-AI Lab & CMU NeuLab for insightful discussions and pointers!
ALT Benchmark correlations (%) with Chatbot Arena Elo, against the total costs of evaluating a single GPT-3.5-Turbo-0125 model. MixEval and MixEval-Hard show the highest correlations with Arena Elo and Arena Elo (En) among leading benchmarks. We reference the crowdsourcing price for Amazon Mechanical Turk ($0.05 per vote) when estimating the cost of evaluating a single model on Chatbot Arena (approximately $2,936). Chatbot Arena is prohibitively expensive, while MixEval and MixEval-Hard are cheap and cost-effective alternatives. For more details, please refer to our paper.
Yifan Song retweeted
๐Excited to share our new paper "LongEmbed: Extending Embedding Models for Long Context Retrieval". We introduce the LongEmbed benchmark, explore context extension of existing embedding models, and release E5-Base-4k & E5-RoPE-Base.
Paper: arxiv.org/abs/2404.12096
Yifan Song retweeted
๐Introducing VisualWebBench: A Comprehensive Benchmark for Multimodal Web Page Understanding and Grounding. visualwebbench.github.io/
๐คWhat's this all about? Why this benchmark?
> Back in Nov 2023, when we released MMMU (mmmu-benchmark.github.io/), a comprehensive multimodal understanding benchmark, we received feedback that it included very few UI screenshots. Considering the growing importance of UI understanding, especially with the rise of powerful agents like Devin (nitter.gq/openaidevs/statuโฆ), which is built on the strong vision capability of #GPT4, we recognized the need for a benchmark focused on UI screenshot understanding.๐ธ๐
> Multimodal #LLMs have significantly boosted web agents' performance on benchmarks like Mind2Web and WebArena. For instance, the SeeAct agent (nitter.gq/ysu_nlp/status/1โฆ) showcases the power of integrating vision into web agents. However, these benchmarks primarily evaluate the end-to-end task execution ability of web agents rather than their understanding of web pages.
๐ Bridging the Gap with VisualWebBench
> To provide a comprehensive evaluation of multimodal LLMs' web page understanding capabilities, we introduce VisualWebBench. Our benchmark spans 139 websites ๐ across 12 domains ๐ท๏ธ and 87 sub-domains ๐, ensuring a diverse and representative dataset. It assesses MLLMs at three levels: website-level, element-level, and action-level ๐, and encompasses seven tasks designed to evaluate understanding, OCR, grounding, and reasoning abilities ๐ง ๐ก.
๐ฎ Surprising Findings
> ๐ Open-source models are catching up: Even though closed-source MLLMs are still leading the leaderboard, we are happy to see open-source models like LLaVA 1.6 34B achieve comparable performance to Gemini Pro.
> ๐ง Grounding ability, crucial for developing MLLM-based web applications, is a weakness for most MLLMs.
> ๐ผ๏ธ Importance of Image Resolution: The limited image resolution handling capabilities of most open-source MLLMs restrict their utility in web scenarios, where rich text and elements are prevalent.
> ๐งฑ Relatively strong correlation with general understanding benchmarks like MMMU but weak correlation with web agent benchmarks like Mind2Web. Web agent benchmarks primarily evaluate the end-to-end task execution ability of web agents, which involves a series of actions to accomplish a goal. In contrast, VisualWebBench emphasizes evaluating the foundational skills of MLLMs such as understanding and grounding web page elements.
๐กFun Fact
> Claude Sonnet is better than Opus on our benchmark :)
๐ Conclusion
> VisualWebBench serves as a valuable resource for the community, driving research and development in the field of multimodal web page understanding and grounding. As MLLMs continue to evolve and improve, we look forward to seeing new applications and breakthroughs. We believe that our benchmark will contribute to the development of more powerful MLLMs in the web domain, ultimately leading to a more intuitive and efficient user experience on the web.
Kudos to the student leads @jeepliu1212 @99Solaris and the team @billyuchenlin, Wai Lam, @gneubig, Yuanzhi Li! ๐
Check out more details in the Junpeng's thread๐
Yifan Song retweeted
Introducing AI2 ๐๐๐๐๐น๐๐๐๐ ! We aim to benchmark LLMs with challenging tasks from real users in the wild. ๐ค Link: hf.co/spaces/allenai/WildBenโฆ
๐คฉ What great features does it offer? ๐x9 โฌ๏ธ
๐1. ๐๐ก๐๐ฅ๐ฅ๐๐ง๐ ๐ข๐ง๐ & ๐๐๐๐ฅ: We carefully curate a collection of 1024 hard tasks from real users, which cover common use cases such as code debugging, creative writing, and data analysis.
๐2. ๐๐ฎ๐ญ๐จ๐๐ฏ๐๐ฅ ๐ฐ/ ๐๐ก๐๐๐ค๐ฅ๐ข๐ฌ๐ญ๐ฌ: Instead of merely asking GPT-4 to choose between A and B, we provide an instance-specific ๐๐ฉ๐ฆ๐ค๐ฌ๐ญ๐ช๐ด๐ต (i.e., a list of evaluation questions) for it to reason before making a judgment. Itโs quite similar to ๐๐ผ๐ง. Thus, our eval is highly interpretable and easy-to-verify.
๐3. ๐๐๐ง๐ ๐ญ๐ก ๐๐๐ง๐๐ฅ๐ญ๐ฒ: GPT-4 judges tend to prefer longer outputs (although humans do too); to avoid this, we devise a simple method to add length penalty on Elo. You can even slide it on our leaderboard UI!
๐4. ๐๐๐ฌ๐ค ๐๐๐ญ๐๐ ๐จ๐ซ๐ข๐ณ๐๐ญ๐ข๐จ๐ง: We tag each example with 12 task types, so we can analyze task-specific performance of LLMs, in addition to their overall ranking.
๐5. ๐
๐๐ข๐ซ ๐๐จ๐ฆ๐ฉ๐๐ซ๐ข๐ฌ๐จ๐ง๐ฌ: WildBench tests all examples on all LLMs. This is different from arena-style evaluation, where one example is only tested on a single pair of models and never seen again.
๐6. ๐๐๐ฌ๐ฒ-๐ญ๐จ-๐ฎ๐ฌ๐: WildBench (v1.0) contains 1024 examples now, and it is extremely easy to add your own LLMs to our leaderboard! We will do the work for you!
๐7. ๐๐ฒ๐ง๐๐ฆ๐ข๐: WildBench will not be a static dataset. We will continue adding new examples and updating evaluation methods based on community feedback.
๐8. ๐๐ฎ๐ฆ๐๐ง ๐๐ฏ๐๐ฅ๐ฎ๐๐ญ๐ข๐จ๐ง (ongoing): We are collecting human preferences via our Leaderboard UI (check the ๐ ๐ tab). Please help us vote! (Weโre planning to recruit domain experts too.)
๐9. ๐๐จ๐ฆ๐ฆ๐ฎ๐ง๐ข๐ญ๐ฒ ๐๐ซ๐ข๐ฏ๐๐ง: We welcome everyone to contribute to human evaluation and create challenging examples. We also value your feedback and suggestions, and will continue enhancing our benchmark leaderboard accordingly.
To sum up, ๐๐๐๐๐น๐๐๐๐ aims to dynamically benchmark LLMs with challenging real-world tasks and provide quick yet reliable evaluations.
Thank you to our amazing team @allen_ai: @khyathi_chandu @faeze_brh @yuntiandeng @lasha_nlp @valentina__py @Ronan_LeBras @YejinChoinka!
Yifan Song retweeted
How does a baby learn to navigate the world around them? ๐ถโโ๏ธ๐ถ Through exploration and learning from each little stumble and triumph. The ETO framework applies this very essence of human learning to AI, emphasizing the importance of both success and failure in developing better AI agents.
ETO has following features:
๐ฒ Learning from Failure Trajectories. Contrary to previous approaches that exclusively train on successful expert trajectories, ETO allows agents to learn from their exploration failures.
๐ญ Contrastive Trajectory Optimization. ETO applies DPO loss to perform policy learning from failure-success trajectory pairs.
๐ Iterative Policy Learning. ETO can be expanded to multiple rounds for further policy enhancement.
โ๏ธ Effectiveness on Three Datasets. ETO significantly outperforms strong baselines, such as RFT, PPO, on WebShop, ScienceWorld, and ALFWorld.
๐ฆพ Generalization on Unseen Scenarios. ETO demonstrates an impressive performance improvement of 22% over SFT on the challenging out-of-distribution test set in ScienceWorld.
Great work led by Yifan Song and @billyuchenlin! Kudos to the team @Wade_Yin9712 @jefffhj!
Check out our work for more details ๐
Trial and Error: Exploration-Based Trajectory Optimization for LLM Agents
Presents an exploration-based trajectory optimization approach, which consistently surpasses baseline performance by a large margin
repo: github.com/Yifan-Song793/ETO
abs: arxiv.org/abs/2403.02502
Obstacles and failures are the most stable stepping stones to success๐ช. This also holds true for LLM agents๐ค.
๐Check out ETO, a framework helps LLM agents learn by trial and error.
๐Great thanks for @Wade_Yin9712, @xiangyue96, @jefffhj, Prof. Sujian Li, and @billyuchenlin!
Trial and Error: Exploration-Based Trajectory Optimization for LLM Agents
Presents an exploration-based trajectory optimization approach, which consistently surpasses baseline performance by a large margin
repo: github.com/Yifan-Song793/ETO
abs: arxiv.org/abs/2403.02502
Yifan Song retweeted
โ๏ธRestGPT
New paper (with code) connecting LLMs with Real-World RESTful APIs by @99Solaris
Code here: github.com/Yifan-Song793/Resโฆ
Always fun to use LangSmith to get a better sense of what's going on under the hood!
Traces:
smith.langchain.com/public/dโฆ
smith.langchain.com/public/aโฆ
๐We just open-sourced RestGPT, an LLM-based autonomous agent controlling real-world applications, such as Spotify and TMDB, via RESTful APIs.
๐คฉMore than 70% success rate on two scenarios!
๐arXiv: arxiv.org/abs/2306.06624
๐Github: github.com/Yifan-Song793/Resโฆ
How to connect large language models with real-world applications?๐ค
๐คฏWe're releasing RestGPT, a framework to connect LLMs with real-world applications, such as Spotify, via RESTful APIs.
Paper: arxiv.org/abs/2306.06624
The code and demo will be released soon.
