Mengsay Loem

Our paper won the Outstanding Paper Award at EMNLP2026! 🏆 OptiMer: Optimal Distribution Vector Merging Is Better than Data Mixing for Continual Pre-Training Haiyue Song, Huy Hien Vu, Masao Utiyama @NICT_Publicity aclanthology.org/2026.emnlp-… Huge thanks to my coauthors, especially @masaoutiyama , for everything during my time at NICT!

✴️ OptiMer is a novel approach to adapt open-source LLMs to your language and your domain, efficiently, effectively, and flexibly! I'll present OptiMer at EMNLP 2026! 📍 Mon Oct 26 · 9:00-10:30 · Room F7 Project Page: shyyhs.github.io/optimer Continual pre-training is widely used to adapt LLMs to target languages and domains, yet the mixture ratio of training data remains a sensitive hyperparameter that is expensive to tune: they must be fixed before training begins, and a suboptimal choice can waste weeks of compute. OptiMer decouples ratio selection from training: 1. we continual pre-train one model per dataset, 2. then extract each model's distribution vector, 3. and search for optimal weights via Bayesian optimization. Results show that OptiMer adapts open-sourced LLMs to languages and domains efficiently, effectively, and flexibly. ⚡15–35× lower search cost than data mixing. 📈 +2.1–6.7 points over data mixing on benchmarks 🔁 Re-optimization for a new language or domain is done in hours without retraining. This work is done with Huy Hien Vu and Masao Utiyama, in the Multilingual and Multicultural Communication Lab, NICT, Kyoto, Japan.