Dan Roth's Cognitive Computation Group at the University of Pennsylvania. (Not the account of the Cognitive Computing Lab at Georgia Tech.)
Philadelphia, PA
Joined May 2009
- Tweets164
- Following66
- Followers660
- Likes98
Cognitive Computation Group retweeted
Earlier today I presented our work on MoNaCo: a QA Benchmark over Dozens of Documents at #ACL2026! Thank you to everyone who attended, and a special thanks to my wonderful co-authors @harsh3vedi @megamor2 @DanRothNLP @yoavgo @DanRothNLP @tusharkhot Ashish Sabharwal @rtsarfaty
Cognitive Computation Group retweeted
Happening Now!!
📌 Decomposed Prompting Does Not Fix Knowledge Gaps, But Helps Models Say “I Don’t Know”
Findings Poster | Presenter: Dhruv Madhwal
🕑 14:00–15:30 | Coronado
A reliability study showing how disagreement across decomposed prompts can signal uncertainty and support abstention. #ACL2026
Cognitive Computation Group retweeted
📷 New #EMNLP2025 Findings survey paper!
“Conflicts in Texts: Data, Implications, and Challenges” Paper: aclanthology.org/2025.findin…
Conflicts are everywhere in NLP — news articles reflecting different perspectives or opposing views, annotators who disagree, LLMs that hallucinate or contradict themselves, and personal/enterprise document collections that grow apart and are conflicting. Most research tackles these in isolation, and our survey provides the first unified view of conflicting information in NLP. We chart the path toward conflict-aware, reliable NLP systems.
Builds on our earlier work on:
- Multi-perspective dataset aclanthology.org/2021.naacl-… and search aclanthology.org/2022.findin…
- Hallucination detection aclanthology.org/2025.findin…
- Open-domain QA with conflicting contexts aclanthology.org/2025.findin…
Cognitive Computation Group retweeted
I will be in #ICML2025 next week and present #ReFocus on Tuesday afternoon.
📍 West Exhibition Hall B2-B3 #W-202
⏱️ Tue 15 Jul 4:30 p.m. PDT - 7 p.m. PDT
Happy to chat and connect! Feel free to DM 😁
ReFocus link: huggingface.co/datasets/ReFo…
Cognitive Computation Group retweeted
#ICLR2025 Oral
LLMs often struggle with reliable and consistent decisions under uncertainty 😵💫 — largely because they can't reliably estimate the probability of each choice.
We propose BIRD 🐦, a framework that significantly enhances LLM decision making under uncertainty.
BIRD = LLM strengths + sound probability theory
BIRD leverages LLMs for world modeling and constructs a Bayesian network using LLM-generated variables, enabling interpretable and trustworthy probability estimates.
✨ BIRD using LLaMA-3.1-70B achieves 30% more accurate probability estimates than GPT-4.
📄 Paper: openreview.net/pdf?id=fAAaT8…
💻 Code: github.com/CogComp/BIRD
Shoutout to my amazing collaborators: @BenZhou96 @wdwlin @DanRothNLP
🗓️ Oral Session 1B: Thu 24 April, 10:54 a.m. – 11:06 a.m. SGT
➡️ Poster: Thu 24 April, 3:00 p.m. – 5:30 p.m. SGT
#AI #LLM #decisionmaking #ProbabilisticAI
(1/n)
Cognitive Computation Group retweeted
Excited to share ✨ Contextualized Evaluations ✨!
Benchmarks like Chatbot Arena contain underspecified queries, which can lead to arbitrary eval judgments. What happens if we provide evaluators with context (e.g who's the user, what's their intent) when judging LM outputs? 🧵↓
Cognitive Computation Group retweeted
Congrats to Peter Baile Chen, Yi Zhang, @DanRothNLP for the Outstanding Paper Award at #ACL2024 Workshop on Knowledgeable LMs!
Is Table Retrieval a Solved Problem? Exploring Join-Aware Multi-Table Retrieval
Peter Baile Chen, Yi Zhang, Dan Roth
arxiv.org/pdf/2404.09889
Tomorrow is the day! We cannot wait to see you at #ACL2024 @aclmeeting Knowledgeable LMs workshop!
Super excited for keynotes by Peter Clark @LukeZettlemoyer @tatsu_hashimoto @IAugenstein @ehovy Hannah Rashkin!
Will announce a Best Paper Award ($500) and a Outstanding Paper Award ($200), kindly supported by @AmazonScience.
Website: knowledgeable-lm.github.io
Location: Lotus Room 10
Zoom: underline.io/events/466/sess…
Meet with our organizers @ZoeyLi20
@megamor2 @eunsolc @mjqzhang @peterbhase @Meng_CS @JiaweiHan @preslav_nakov @mohitban47 @hengjinlp! Looking forward to talking with you!
#LLMs #knowledge #NLProc
Cognitive Computation Group retweeted
Cognitive Computation Group retweeted
Excited to share new work done @GoogleDeepMind: 🏔️ DOLOMITES: Domain-Specific Long-Form Methodical Tasks, a new long-form generation benchmark for evaluating language models on **realistic** domain-specific tasks.
Website: dolomites-benchmark.github.i…
Paper: arxiv.org/abs/2405.05938
Cognitive Computation Group retweeted
Our workshop, AI4Research (ai4research.github.io/), is looking for reviewers who have expertise in areas such as AI4Material, AI4Medical, AI for Single-Cell, and AI4Education. Please let us know if you are interested.
Cognitive Computation Group retweeted
Can GPT-4V and Gemini-Pro perceive the world the way humans do? 🤔
Can they solve the vision tasks that humans can in the blink of an eye? 😉
tldr; NO, they are far worse than us 💁🏻♀️
Introducing BLINK👁 zeyofu.github.io/blink/, a novel benchmark that studies visual perception abilities NOT yet “emerged” in Multimodal LLMs 🔥🔥
Paper: arxiv.org/abs/2404.12390
(1/n)
Cognitive Computation Group retweeted
During a long-chained reasoning process...
🤔Are LLMs following correct reasoning paths or relying on semantic shortcuts?
😵💫 What contributes to the failure of LLMs?
🛒Will prompting/RAG solve it?
Check out our paper, which is accepted by #NAACL2024 at: vincentleebang.github.io/eur…
Cognitive Computation Group retweeted
We are pleased to announce that the first Conference on Language Modeling will be held at the University of Pennsylvania in Philadelphia at the Zellerbach Theatre.
Thanks so much to UPenn CS as well as Mark Yatskar and Zachary Ives for facilitating the amazing venue.
Cognitive Computation Group retweeted
❗In RAG setting, the unit in which you segment a retrieval corpus (passage, 100-word chunks, etc) may impact performance by more than you thought!
Preprint: arxiv.org/abs/2312.06648
Led by @tomchen0.
Github: github.com/chentong0/fact…
Demo: chentong0.github.io/factoid-…
🎉𝐃𝐞𝐧𝐬𝐞 𝐗 𝐑𝐞𝐭𝐫𝐢𝐞𝐯𝐚𝐥: What Retrieval Granularity Should We Use?
Both passage and sentence level index are not optimal for dense retrieval. We introduce a novel retrieval unit, proposition, for dense retrieval. See details in this thread ~
arxiv.org/abs/2312.06648
Cognitive Computation Group retweeted
🚀 Excited to share our #EMNLP2023 (#NLP) paper on "TempTabQA: Temporal Question Answering for Semi-Structured Tables"!
web: temptab.github.io
📚🔍 We explore temporal question answering across semi-structured tabular data and introduce the TempTabQA dataset. #NLProc 1/n
Cognitive Computation Group retweeted
Excited to share 🎰 Pachinko: Patching Interpretable QA Models through Natural Language Feedback
chaitanyamalaviya.github.io/…
Work done w/ @subinlee_ @danrothnlp @yatskar
Cognitive Computation Group retweeted
(1/12) Excited to share 🩺 ExpertQA ⚖️ : a new benchmark for attribution & factuality analysis, as well as long-form QA, created with experts from 32 fields!
arxiv.org/abs/2309.07852
Work done @Penn & @UWMedicine, with @subinlee_ @soshsihao @Esieber17 @yatskar @DanRothNLP.
Cognitive Computation Group retweeted
Wondering how to split complex sentences into simpler "atomic" forms? Check out our work to appear in #ACL2023NLP findings!
We recently uploaded a processed, more friendly version of the dataset on huggingface hub, check it out!
huggingface.co/datasets/siha…
📢 New Dataset: PropSegmEnt 📢
Can we build models to recognize if/which parts of a sentence are true? Check out our new dataset with >35K human annotated propositions with fine-grained NLI/textual entailment labels!
🔗: github.com/google-research-d…
📄: github.com/google-research-d…
1/N
Cognitive Computation Group retweeted
Can language models including GPT-3.5 and GPT-4 perform complex reasoning tasks, e.g., temporal reasoning, with the “right reasons”? ⛔️There’s still much room for improvement.
Check out our new paper at ACL 2023 📜: arxiv.org/abs/2212.10467
(1/N)
