ML, NLP, Quant research. Graduate from @ntunlpsg | @AstarI2r | @uoftcompsci | @centralesupelec. Currently QR at ADIA.
United Arab Emirates
Joined August 2014
- Tweets190
- Following2.6K
- Followers426
- Likes2K
Mathieu Ravaut retweeted
As large language models increasingly serve as judges for evaluating other models during both development and deployment, most existing benchmarks still focus on non-contextual tasks like chat completions or logical reasoning. Our team present ⚖️ ContextualJudgeBench ⚖️ — a benchmark designed to evaluate LLM-as-judge capabilities in context-rich scenarios such as RAG-based QA and summarization.
In real-world business applications, context is everything—AI systems must provide accurate and timely information. But assessing their performance is challenging: judge models need to interpret both the context and the question, then evaluate accuracy, justification for refusals, and overall quality, including completeness and conciseness.
📘 Paper: arxiv.org/abs/2503.15620
📊 Data: huggingface.co/datasets/Sale…
💻 Code: github.com/SalesforceAIResea…
Mathieu Ravaut retweeted
Testing LLMs' reasoning skills is tough—human evaluations are expensive, data contamination is common, and LLM judges can be biased. We propose StructTest, the first benchmark that checks how well LLMs follow complex instructions and create structured outputs. It uses a rule-based evaluator that’s easy to adapt to new tasks. StructTest is unbiased, cheap, hard to cheat and highly scalable.
By testing structured outputs in areas like Summarization, Code, HTML, and Math—and evaluating 17 top LLMs—StructTest proves to be a challenge even for models like Deepseek-V3/R1 and GPT-4o. It’s also highly correlated with ChatBot Arena (~93%) and MMLU (>96%), making it a solid way to measure reasoning skills.
Code & Data: github.com/SparkJiao/StructT…
Paper🔗: arxiv.org/abs/2412.18011
Checkout our analysis paper, to be featured in ACL main conference:
arxiv.org/abs/2310.10570
We investigate the "middle-curse" exposed by the LITM paper for the specific task of abstractive summarizarion. In brief: the middle curse is very much an issue ! w @JotyShafiq @AixinSG – at Central Region, Singapore
Mathieu Ravaut retweeted
Hope this paper and the accompanying library become handy to the LLM community.
Are your LLMs highly accurate, or simply contaminated?
As the race to build the best LLM intensifies, clean evaluation is becoming more important than ever, yet contaminated LLMs and benchmarks obfuscate the real performance of models.
Checkout our new work (comprehensive survey + library) at NTU-NLP lab & Salesforce Research on the critical issue of contamination detection in LLMs, cc @ntunlp @MatRavox @D_Boss001 @HailinChen3 @XingxuanLi @RuochenZhao3 @FangkaiJiao @qcwntu @CaimingXiong @JotyShafiq
Paper:
arxiv.org/abs/2404.00699
Library:
github.com/ntunlp/LLMSanitiz…
Mathieu Ravaut retweeted
Are your LLMs highly accurate, or simply contaminated?
As the race to build the best LLM intensifies, clean evaluation is becoming more important than ever, yet contaminated LLMs and benchmarks obfuscate the real performance of models.
Checkout our new work (comprehensive survey + library) at NTU-NLP lab & Salesforce Research on the critical issue of contamination detection in LLMs, cc @ntunlp @MatRavox @D_Boss001 @HailinChen3 @XingxuanLi @RuochenZhao3 @FangkaiJiao @qcwntu @CaimingXiong @JotyShafiq
Paper:
arxiv.org/abs/2404.00699
Library:
github.com/ntunlp/LLMSanitiz…
Mathieu Ravaut retweeted
We are excited to share that some of our Criteos got the "Best Paper Award" at #EACL2024 👏🎉
Congrats @florian_flb @songdng @MatRavox @AlexAllauzen , Nancy F. Chen,
@albertlumbreras @LaureSoulier, Vincent Guigue and Patrick Gallinari.🏆
@eaclmeeting
📜 arxiv.org/abs/2401.17919
Checkout our paper on SSMs for long-input summarization! Happening today at EACL 👇
Congrats to @florian_flb @songdng
🤔 Ever wondered how to summarize an entire book 𝑎𝑙𝑙 𝑎𝑡 𝑜𝑛𝑐𝑒? We delve into this question in our recent paper “𝐋𝐎𝐂𝐎𝐒𝐓: 𝐒𝐭𝐚𝐭𝐞-𝐒𝐩𝐚𝐜𝐞 𝐌𝐨𝐝𝐞𝐥𝐬 𝐟𝐨𝐫 𝐋𝐨𝐧𝐠 𝐃𝐨𝐜𝐮𝐦𝐞𝐧𝐭 𝐀𝐛𝐬𝐭𝐫𝐚𝐜𝐭𝐢𝐯𝐞 𝐒𝐮𝐦𝐦𝐚𝐫𝐢𝐳𝐚𝐭𝐢𝐨𝐧” (#EACL2024). 🧵 (1/8)
Ever wondered how to summarize an entire book *all at once* with a deep learning model?
Check out our latest paper on summarization:
arxiv.org/abs/2401.17919
Through State Space models, we can scale the input to 600k tokens (!)
Mathieu Ravaut retweeted
We are presenting a tutorial on NLP+Vis: NLP Meets Visualization @emnlp this morning at 9am local time. Explore more at nlp4vis.github.io. #EMNLP2023
@JotyShafiq @jesse_vig
Check our survey - ChatGPT is gradually getting dethroned by open source alternatives
It has been exactly one year since the release of ChatGPT. How far are open-source LLMs? We provide an exhaustive review of open-source LLMs that claim to catch up with or surpass ChatGPT in various capabilities.
paper🔗: arxiv.org/pdf/2311.16989.pdf
🧵(1/5)
Mathieu Ravaut retweeted
Open-Source LLMs vs. ChatGPT:
1. General Capabilities: Llama-2-chat-70B variant exhibits enhanced capabilities in general conversational tasks, surpassing the performance of GPT-3.5-turbo; UltraLlama matches GPT-3.5-turbo’s performance in its proposed benchmark.
2. Agent Capabilities (using tools, self-debugging, following natural language feedback, exploring environment): Lemur-70B-chat surpasses the performance of GPT-3.5-turbo when exploring the environment or following natural language feedback on coding tasks. AgentLlama-70B achieves comparable performance to GPT-3.5-turbo on unseen agent tasks. Gorilla outperforms GPT-4 on writing API calls.
3. Logical Reasoning Capabilities: fine-tuned models (e.g., WizardCoder, WizardMath) and pre-training on higher quality data models (e.g., Lemur-70B-chat, Phi-1, Phi-1.5) show stronger performance than GPT-3.5-turbo.
4. Modeling Long-Context Capabilities: Llama-2-long-chat-70B outperforms GPT-3.5-turbo-16k on ZeroSCROLLS.
5. Application-specific Capabilities:
- query-focused summarization (fine-tuning on training data is better)
- open-ended QA (InstructRetro shows improvement over GPT3)
- medical (MentalLlama-chat-13 and Radiology-Llama-2 outperform ChatGPT)
- generate structured responses (Struc-Bench outperforms ChatGPT)
- generate critiques (Shepherd is almost on-par with ChatGPT)
6. Trust-worthy AI:
- hallucination: during finetuning - improving data quality during fine-tuning; during inference - specific decoding strategies, external knowledge augmentation (Chain-of-Knowledge, LLM-AUGMENTER, Knowledge Solver, CRITIC, Prametric Knowlege Guiding), and multi-agent dialogue.
- safety: GPT-3.5-turbo and GPT-4 models remain at the top for safety evaluations. This is largely attributed to Reinforcement Learning with Human Feedback (RLHF). RL from AI Feedback (RLAIF) could help reduce costs for RLHF.
🔗arxiv.org/abs/2311.16989
Thanks to the authors for the great paper! @CaimingXiong @HailinChen3 @FangkaiJiao @qcwntu
@XingxuanLi @RuochenZhao3 @MatRavox @JotyShafiq
Mathieu Ravaut retweeted
It has been exactly one year since the release of #ChatGPT. How far are open-source LLMs? We provide an exhaustive review of open-source LLMs that claim to catch up with or surpass ChatGPT in various capabilities.
paper🔗: arxiv.org/pdf/2311.16989.pdf
Mathieu Ravaut retweeted
Altman, Hassabis, and Amodei are the ones doing massive corporate lobbying at the moment.
They are the ones who are attempting to perform a regulatory capture of the AI industry.
You, Geoff, and Yoshua are giving ammunition to those who are lobbying for a ban on open AI R&D.
If your fear-mongering campaigns succeed, they will *inevitably* result in what you and I would identify as a catastrophe: a small number of companies will control AI.
The vast majority of our academic colleagues are massively in favor of open AI R&D. Very few believe in the doomsday scenarios you have promoted.
You, Yoshua, Geoff, and Stuart are the singular-but-vocal exceptions.
like many, I very much support open AI platforms because I believe in a combination of forces: people's creativity, democracy, market forces, and product regulations.
I also know that producing AI systems that are safe and under our control is possible. I've made concrete proposals to that effect.
This will all drive people to do the Right Thing.
You write as if AI is just happening, as if it were some natural phenomenon beyond our control.
But it's not. It's making progress because of individual people that you and I know. We, and they, have agency in building the Right Things.
Asking for regulation of R&D (as opposed to product deployment) implicitly assumes that these people and the organization they work for are incompetent, reckless, self-destructive, or evil. They are not.
I have made lots of arguments that the doomsday scenarios you are so afraid of are preposterous. I'm not going to repeat them here. But the main point is that if powerful AI systems are driven by objectives (which include guardrails) they will be safe and controllable because *e* set those guardrails and objectives.
(Current Auto-Regressive LLMs are not driven by objectives, so let's not extrapolate from their current weaknesses).
Now about open source: your campaign is going to have the exact opposite effect of what you seek.
In a future where AI systems are poised to constitute the repository of all human knowledge and culture, we *need* the platforms to be open source and freely available so that everyone can contribute to them.
Openness is the only way to make AI platforms reflect the entirety of human knowledge and culture.
This requires that contributions to those platforms be crowd-sourced, a bit like Wikipedia.
That won't work unless the platforms are open.
The alternative, which will *inevitably* happen if open source AI is regulated out of existence, is that a small number of companies from the West Coast of the US and China will control AI platform and hence control people's entire digital diet.
What does that mean for democracy?
What does that mean for cultural diversity?
*THIS* is what keeps me up at night.
Checkout our latest work on summarization!
Working paper: LLMs prioritize the introductory section of lengthy documents when generating summaries.
arxiv.org/abs/2310.10570
A related question: what do LLMs really gain from an extremely long context? Is there some form of "summarization" when taking in the long context?
Mathieu Ravaut retweeted
Learned a lot about LLM training while co-leading this work. Finally it is out. Check it out.
🔥Introducing XGen-7B, a new 7B LLM trained on 8K seq. length for 1.5T tokens. Better or comparable results with MPT, Falcon, LLaMA, OpenLLaMA in text & code tasks.
Blog: blog.salesforceairesearch.co…
Code: github.com/salesforce/xgen
Training cost ~$150K for 1T token
Mathieu Ravaut retweeted
🔥A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasets🔥
Yet another ChatGPT evaluation !!!! What's new ?
This time not automatic, human in a loop.
arxiv.org/pdf/2305.18486.pdf
Our ACL'23 paper covers FULL eval on benchmarks that actually MATTERS !!!
Not happy about the abstractive summaries that you are generating from the language model?
Have a try at sampling many and re-ranking them! 👇
arxiv.org/abs/2212.09593 – at Central Region, Singapore
Re-ranked sunmaries are more factually consistent, due to enhancing similarity with the source document.