Measuring and forecasting AI progress @AnthropicAI.
San Francisco, CA
Joined March 2011
- Tweets479
- Following529
- Followers4.5K
- Likes5.1K
Pinned Tweet
AI systems often use more direct experience than a human could get in a lifetime. Humans require less experience, because they "transfer" past experience to new tasks. Recent work I led found an equation to characterize transfer in a simple setting.
arxiv.org/pdf/2102.01293.pdf
👇
Danny Hernandez retweeted
Today, AI can generate tons of code—but how do we know if it's good?
That's why we built Sculptor: the first coding agent environment.
Sculptor helps you catch issues, write tests, and improve your code—all while you work in your favorite editor.
Danny Hernandez retweeted
Machines of Loving Grace: my essay on how AI could transform the world for the better
darioamodei.com/machines-of-…
Danny Hernandez retweeted
Introducing Claude for Enterprise.
Now your entire organization can collaborate securely with Claude—with no training on chats or files.
Comes with:
📚 Expanded 500K context window
🧑💻 Native GitHub integration
🔐 Enterprise-grade security features
anthropic.com/news/claude-fo…
Danny Hernandez retweeted
A big part of my job these days is to think about what technical work Anthropic needs to do to make things go well with the development of very powerful AI.
I digested my thinking on this, plus some of the Anthropic zeitgeist around it, into this piece:
sleepinyourhat.github.io/che…
ALT A beautiful sunset over calm seas, but with waves, rocks, and a rusty concrete staircase in the foreground. Credit: Justin Kern via flickr at https://www.flickr.com/photos/justinwkern/5972229589
Danny Hernandez retweeted
I'm excited to join @AnthropicAI to continue the superalignment mission!
My new team will work on scalable oversight, weak-to-strong generalization, and automated alignment research.
If you're interested in joining, my dms are open.
Danny Hernandez retweeted
New Anthropic research paper: Scaling Monosemanticity.
The first ever detailed look inside a leading large language model.
Read the blog post here: anthropic.com/research/mappi…
ALT Title card for Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet
Danny Hernandez retweeted
The word hasn't gotten out yet just how good Claude 3 Opus is for economics and economic reasoning. So here's the word.
I find our latest model to have a lot more nuanced and careful thinking. Feels like a big step towards the thought partner I want.
Today, we're announcing Claude 3, our next generation of AI models.
The three state-of-the-art models—Claude 3 Opus, Claude 3 Sonnet, and Claude 3 Haiku—set new industry benchmarks across reasoning, math, coding, multilingual understanding, and vision.
ALT A table of Claude 3 model family benchmarks. Claude 3 Opus, the most capable model, exceeds SOTA across reasoning, math, code, and other evaluations versus GPT-4 and Gemini Ultra.
Danny Hernandez retweeted
New Anthropic Paper: Sleeper Agents.
We trained LLMs to act secretly malicious. We found that, despite our best efforts at alignment training, deception still slipped through.
arxiv.org/abs/2401.05566
ALT Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training
Danny Hernandez retweeted
Our new model Claude 2.1 offers an industry-leading 200K token context window, a 2x decrease in hallucination rates, system prompts, tool use, and updated pricing.
Claude 2.1 is available over API in our Console, and is powering our claude.ai chat experience.
Danny Hernandez retweeted
Today, we’re publishing our Responsible Scaling Policy (RSP) – a series of technical and organizational protocols to help us manage the risks of developing increasingly capable AI systems.
ALT ASL-1: Smaller models. ASL-2: Present large models. ASL-3: Significantly higher risk. ASL-4+: Speculative. Increasing model capability, Increasing security and safety measures.
We should expect a jump that in some sense feels like the jump between gpt3 and claude/gpt4 in the next ~2 years, based on smooth underlying exponentials in effective compute.
There is lots of meaning in trying to make that go well and meaning is top of the hierarchy of needs.
How quickly is A.I. advancing? And should you be working in the field? Checkout my recent conversation on these topics with @Hernandez_Danny:
podcast.clearerthinking.org/…
Danny Hernandez retweeted
Introducing Claude 2! Our latest model has improved performance in coding, math and reasoning. It can produce longer responses, and is available in a new public-facing beta website at claude.ai in the US and UK.
ALT Claude 2's performance on the GRE, USMLE, and Multistate Bar Exam. Claude 2 on GRE: Verbal: 165; Analytical Writing: 5.0; Quant Reasoning: 154. Claude 2 on USMLE: Step 1 (5-shot) 68.9; Step 2: 63.3; Step 3: 67.2. Claude 2 on Multistate Bar Exam (5-shot): 76.5
Danny Hernandez retweeted
We develop a method to test global opinions represented in language models. We find the opinions represented by the models are most similar to those of the participants in USA, Canada, and some European countries. We also show the responses are steerable in separate experiments.
Danny Hernandez retweeted
We're introducing the first AI-based live global weather forecast.
Available to everyone at earth.atmo.ai
Danny Hernandez retweeted
Introducing 100K Context Windows! We’ve expanded Claude’s context window to 100,000 tokens of text, corresponding to around 75K words. Submit hundreds of pages of materials for Claude to digest and analyze. Conversations with Claude can go on for hours or days.
Danny Hernandez retweeted
Proud to partner with @awscloud to give people an easy way to access Claude in their cloud environments!
We believe this work will drive immense value for businesses looking to build generative AI applications with AWS tools and capabilities.
Stay tuned for more.
Deep learning. Large language models. Vast capabilities. ☁️💻💡
From chatbots to code generation, learn how #GenerativeAI is redefining #ML-powered capabilities—& how you can build & use large language on #AWS. #MachineLearning #AI
👉 go.aws/416RimS
General access to Claude via slack.
Today we are releasing the new Claude App for @SlackHQ, in beta. Now every company in the world has the chance to have a “virtual teammate” who can help make work more fun and productive.
Only model I'm aware of comparable to ChatGPT.
Have some notable customers. Some prefer its personality, clarity, summarization, creativity, etc. I Makes sense for any business using ChatGPT to compare it to Claude.
After working for the past few moths with key partners like @NotionHQ, @Quora, and @DuckDuckGo, we’ve been able to carefully test out our systems in the wild. We are now opening up access to Claude, our AI assistant, to power businesses at scale.
Writing AI evals is the first AI researcher task I've seen where models feel clearly better than me.
This automation will enable much better measurement and understanding of LM's broadly.
Reduces eval iteration time from ~week to ~hour.
>10x more leverage from a researcher.
It’s hard work to make evaluations for language models (LMs). We’ve developed an automated way to generate evaluations with LMs, significantly reducing the effort involved. We test LMs using >150 LM-written evaluations, uncovering novel LM behaviors.
anthropic.com/model-written-…
