cs/ml @ sword health, técnico lisbon
Joined August 2023
- Tweets17
- Following469
- Followers26
- Likes2.8K
hugo pitorro retweeted
Sword Health built Dawn 1 on prime-rl:
They RL-post-trained GLM-5.3 into a mental health support model that is safer, less sycophantic, and more cost-efficient than its base
Introducing Dawn 1, Sword's frontier model for mental health support.
Post-trained from GLM-5.3, Dawn 1 delivers greater intelligence at lower inference cost, with strong performance across mental health knowledge, safety, and sycophancy evaluations. It also outperforms frontier proprietary models on multi-turn safety protocol adherence in our evaluations.
A look at the research 🧵
hugo pitorro retweeted
Introducing Dawn 1, Sword's frontier model for mental health support.
Post-trained from GLM-5.3, Dawn 1 delivers greater intelligence at lower inference cost, with strong performance across mental health knowledge, safety, and sycophancy evaluations. It also outperforms frontier proprietary models on multi-turn safety protocol adherence in our evaluations.
A look at the research 🧵
hugo pitorro retweeted
Glad to share that AdaSplash-2 was accepted to ICML 2026! 🇰🇷
Btw, we've released the Triton kernels for AdaSplash-2 in our GitHub repo. Nuno's histogram implementation with bit-packing to avoid atomics is, in my totally unbiased opinion, remarkable!
github.com/deep-spin/adaspla…
hugo pitorro retweeted
We have just released AdaSplash 2, a highly efficient implementation of adaptively sparse attention!
- Faster than FlashAttention 2 during training when block sparsity > 60%
- More accurate than softmax attention on long-context benchmarks (+16 on HELMET ICL at 32k length)!
hugo pitorro retweeted
Markets down 📉 NVIDIA LLM research 📈 Nemotron-H: A Family of Accurate and Efficient Hybrid Mamba-Transformer Models!
Better or on-par accuracy compared to other similar open-sourced transformer models while being up to ✨ 3× faster at inference ✨
arxiv.org/abs/2504.03624
New paper with @MarcosTreviso! We introduce LaTIM: a token-decomposition method for Mamba models allowing us to capture precise attention maps similar to transformers.
Paper: arxiv.org/abs/2502.15612
Code: github.com/deep-spin/latim
hugo pitorro retweeted
This is interesting. Decoder-only don't beat Enc-Decoder when similar in params and same training data for #neuralempty
Ultimately Mamba can work (but somehow learning data distribution > language generalization).
Nice work @andre_t_martins et al.
arxiv.org/pdf/2407.05489
hugo pitorro retweeted
Nous Research announces the pre-training of a 15B parameter language model over the internet, using Nous DisTrO and heterogeneous hardware contributed by our partners at @Oracle, @LambdaAPI, @NorthernDataGrp, @CrusoeCloud, and the Andromeda Cluster.
This run presents a loss curve and convergence rate that meets or exceeds centralized training.
Our paper and code on DeMo, the foundational research that led to Nous DisTrO, is now available (linked below).
hugo pitorro retweeted
State space models have struggled to learn to do things like copying and associative recall 🟢 -- things that self-attention learns easily 🟠...
But it turns out we just needed to change SSM initialization a bit 🔵. Our init helps a lot, and even makes state space layers *look* more like self-attention layers.
Maybe SSMs have been underestimated? Read more: arxiv.org/abs/2410.11135
w/ @harhrayr, @zicokolter, Sanjiv Kumar, Srinadh Bhojanapalli
hugo pitorro retweeted
Ever wanted to scale subquadratic models up to 7B+ LLMs? But didn't want to pretrain billions of parameters on trillions of tokens?
Then just for you, we're happy to share LoLCATs 😺
We show how to convert existing Transformers like Llama 3 8B & Mistral 7B into state-of-the-art subquadratic LLMs
But now for the same cost as a LoRA finetune! (Train just 0.2% of an LLM's parameters, on 0.04-0.2% of prior conversion methods’ training tokens; 0.003% that of pretraining from scratch)
And then kind of just because we could, used LoLCATs to make the first linearized 70B and 405B LLMs (🙏 @AIatMeta for Llama 3.1)
Preprint: github.com/HazyResearch/lolc…
Code: github.com/HazyResearch/lolc…
Blog : hazyresearch.stanford.edu/bl…
and more in 🧵 below, so you can too!
hugo pitorro retweeted
🚀 Just published: "Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Experts" 🌐
Introducing Loss-Free Balancing—our latest innovation in MoE models that ditches the need for auxiliary loss. By dynamically adjusting expert biases, we ensure optimal load balance without the side effects of unwanted gradients. Validated on models up to 3B parameters, our approach delivers better validation loss and load balance than traditional methods.
Technical report: arxiv.org/abs/2408.15664
