Pinned Tweet
Can LLMs accurately aggregate information over long, information-dense texts? Not yetโฆ
We introduce Oolong, a dataset of simple-to-verify information aggregation questions over long inputs. No model achieves >50% accuracy at 128K on Oolong!
ALT Performance of a sweep of models on Oolong-synth and Oolong-real. Performance decreases with increasing context length, sometimes steeply.
Amanda Bertsch @COLM retweeted
The UT comp ling faculty (@jessyjli, @kanishkamisra, @kmahowald & myself) are looking for PhD students to join us! sites.utexas.edu/compling/ ๐ค
We are an interdisciplinary group with interests across linguistics, cogsci and NLP. Apply by 12/1: liberalarts.utexas.edu/linguโฆ
Amanda Bertsch @COLM retweeted
Honoured to receive an outstanding paper award for our work! Couldnโt have done it without all my coauthors @KaiserWhoLearns @millicent_li @wordscompute @lltjuatja @JentseHuang and my advisor @gneubig ๐ซก
Congratulations to the authors of the 3 outstanding papers!
arxiv.org/abs/2607.01077
arxiv.org/abs/2604.08510
arxiv.org/abs/2604.13977
I'm at #COLM2026 and on the academic job market this year! I'd love to chat about long context reasoning, distilling harness capabilities into architectural improvements, and better training methods for complex corpus tasks.
I'll be presenting work on Tuesday + Thursday โฌ๏ธ
On Tuesday, from 11-1, I'll be presenting Oolong, our benchmark on long-context distributional reasoning. Poster #87 in Franciscan B! xitter.cf/abertsch72/status/1986โฆ
Can LLMs accurately aggregate information over long, information-dense texts? Not yetโฆ
We introduce Oolong, a dataset of simple-to-verify information aggregation questions over long inputs. No model achieves >50% accuracy at 128K on Oolong!
ALT Performance of a sweep of models on Oolong-synth and Oolong-real. Performance decreases with increasing context length, sometimes steeply.
And on Thursday, 4:40-6:30, I'll be presenting Olmpool, our data-controlled study of how small architectural variations impact the ability to generalize to longer contexts! Poster #3 in the Imperial Ballroom. xitter.cf/abertsch72/status/2049โฆ
New paper! allenai.org/papers/olmpool
This tackles a puzzle we found during the training of Olmo 3: how could two models with nearly identical short-context performance (and trained on the same data!) behave completely differently after long context extension?
Amanda Bertsch @COLM retweeted
Excited to be presenting my GDM internship work on cultural alignment/interp at COLM next week!
Room: Franciscan B; Poster Location: #81 at 11:00 AM on Tuesday, October 6th!
Would love to meet old friends and new :') Happy to chat about anything, been working on AI alignment, evals, post-training, image generation; but broadly also been thinking about domain applications esp. healthcare & education!
I'm also starting my industry job search and would be happy to connect with people for relevant opportunities if I'd be a good fit :)
Website -- simran-khanuja.github.io/
Which cultures do LLMs best align with on open-ended questions? Why? Can we change that default?
We study this in our new work: "Steering LLMs for Culturally Localized Generation" (COLM 2026), from my time at Google DeepMind! ๐งต
arxiv.org/abs/2603.23301
What makes a task "hard"? We study one notion of difficulty: how much cross-context comparison is necessary to solve a task. Tasks with high CTC show tradeoffs between architectures that don't exist for traditional retrieval or short-context tasks!
Could long-context architectures โfind all contradictionsโ in a science literature? Not yet! ๐งต
We study a new class of "high-complexityโ tasks whose difficulty scales quadratically with corpus size (as opposed to linearly), reversing common LCLM decisions! (block-sparse attention, hybrid modelsโฆ)
Amanda Bertsch @COLM retweeted
๐ขI am hiring interns at the Max Planck Institute!
Topics of interest:
(1) LLM factuality, reliable information synthesis and reasoning (esp. in agentic settings), personalization
(2) AI creativity
(3) Data-centric interpretability
Apply here by Sep 15: shorturl.at/RmSb1
Amanda Bertsch @COLM retweeted
๐จ๐ฆ๐๐ฏ๐บ๐ถ๐๐๐ถ๐ผ๐ป ๐ฑ๐ฒ๐ฎ๐ฑ๐น๐ถ๐ป๐ฒ ๐ฒ๐
๐๐ฒ๐ป๐ฑ๐ฒ๐ฑ ๐๐ผ ๐ฆ๐ฒ๐ฝ๐. ๐ญ๐ฏ, ๐ฎ๐ฌ๐ฎ๐ฒ (๐๐ผ๐)
Weโre excited to welcome more submissions to the ๐๐ผ๐ป๐ด-๐๐ผ๐ป๐๐ฒ๐
๐ ๐๐ผ๐๐ป๐ฑ๐ฎ๐๐ถ๐ผ๐ป ๐ ๐ผ๐ฑ๐ฒ๐น๐ ๐ช๐ผ๐ฟ๐ธ๐๐ต๐ผ๐ฝ @ ๐ก๐ฒ๐๐ฟ๐๐ฃ๐ฆ ๐ฎ๐ฌ๐ฎ๐ฒ in ๐๐๐น๐ฎ๐ป๐๐ฎ!
๐ OpenReview Link: bit.ly/3S0BLpG
๐ Interested in reviewing? Please fill out the form: forms.gle/wsXwirySry7aymHD6
Topics include, but are not limited to:
๐ค Long-context & long-horizon agents
๐ง Novel modeling & training
โก Efficient foundation models
๐ Robust evaluation
๐ Long-context reasoning
๐ฅ Long-context multimodal learning
๐ฌ Long-horizon AI for science
More details in ๐ longcontextfm.github.io
Feel free to reach out to our team if you have any questions: @abertsch72 @HowardYen1 @SeungjuHan3
Are you working on long context modeling, long horizon agents, or any other situation where you're tackling large amount of data (text, image, video, speech, biology....) with LLMs? Submit to our NeurIPS Atlanta workshop by September 10th!
OpenReview: bit.ly/3S0BLpG
This Post is from an account that no longer exists. Learn more
Amanda Bertsch @COLM retweeted
I will be at #ACL2026 from July 2-7! Along with checking out the conference and hanging around SD, I'm also โจrecruiting students for Fall 2027 @ UT Austin Linguistics (along with @kanishkamisra)โจ
Some topics I'm really excited about below โฌ๏ธ
Amanda Bertsch @COLM retweeted
As my time in Pittsburgh comes to an end...
I'm excited to share that I will be joining UIUC as an Assistant Professor in fall 2027! I will recruit PhD students in both ECE and CS.
Iโm looking for students and postdocs who are excited to continue building towards collaborative AI systems that learn and adapt through interaction.
Amanda Bertsch @COLM retweeted
In case youโve been wondering what Iโve been up to these daysโฆ
So excited to (re)join an amazing community of linguists and NLP researchers at UT :)
We are excited to announce that Lindia Tjuatja (@lltjuatja) will be joining us as an Assistant Professor,ย starting in Fall 2027! Lindia is an alum of UT Linguistics and Electrical and Computer Engineering,ย and is currently finishing her PhD at CMU. Welcome back to UT, Lindia!
Amanda Bertsch @COLM retweeted
post-trained models are more helpful, but collapse toward a narrow range of possible answers
๐ with ReDiPO, we show how to recover the lost diversity with a simple DPO data pipeline, while largely preserving instruction-following and safety
great work led by @vsamuel2003 !
Amanda Bertsch @COLM retweeted
Sub-agents are a promising inference-time scaling primitive:
โข Expand an agent's working memory
โข Divide-and-conquer hard problems
โข Solve problems faster with parallel execution
But how do we train a model to best take advantage of sub-agents and make sure we get these benefits?
Very excited to release RAO: Recursive Agent Optimization.
RAO is an end-to-end reinforcement learning approach for training LLM agents to spawn, delegate to, and coordinate with recursive copies of themselves (that can themselves spawn other agents) - turning recursive inference into a learned capability.
1/10
New paper! allenai.org/papers/olmpool
This tackles a puzzle we found during the training of Olmo 3: how could two models with nearly identical short-context performance (and trained on the same data!) behave completely differently after long context extension?
Check out the paper for much more analysis, including estimating long context performance from short context (really hard!), additional pretraining settings that DON'T matter for long context (float8 linear layers!), and analysis of attention distributions for each model.
This was an absolute joy of a project with wonderful mentors at Ai2: @mechanicaldirk @soldni @kylelostat @HannaHajishirzi (plus @gneubig + Matt Gormley!)
Paper: allenai.org/papers/olmpool
Models: huggingface.co/collections/aโฆ