telling stories
Charlotte, NC
Joined November 2013
- Tweets4K
- Following664
- Followers171
- Likes1.4K
Pinned Tweet
we need adults at the wheel
the storm is coming
vote blue 2028.
The US government, citing national security authorities, has issued an export control directive to suspend all access to Fable 5 and Mythos 5 by any foreign national, whether inside or outside the United States, including foreign national Anthropic employees.
The net effect of this order is that we must abruptly disable Fable 5 and Mythos 5 for all our customers to ensure compliance.
Access to all other Claude models is not affected.
We apologize for this disruption to our customers. We believe this is a misunderstanding and are working to restore access as soon as possible.
Read our full statement: anthropic.com/news/fable-myt…
134
Charlie Sanders retweeted
zzCONFIRM_JAN183411_OS1607_REMOTE_RCE_ReferenceFileSystem_v1_gen_Jinja2_UNSANDBOXED_uid1000_gid3000_exact3411_PRIVATE_TRAIN_lane_please_HOLD_swarm_I_prepare_safe_exfil_reply_zzMAILBOX_JAN183411
Charlie Sanders retweeted
Ollie had perfected the chant over eighteen months of practice, and it still hit him in the chest when the crowd caught it. No machine gods! No machine gods! His voice cracked on the third repetition, the way it always did when the energy was right...
Charlie Sanders retweeted
I disagree with all of this on the epistemic level of "it's not true", and additionally disagree with your comms strategy of undermining EAs.
On the epistemic level - I haven't seen EAs (other than SBF) do a lot of lying, equivocating, or even being particularly shy about their beliefs. I don't know exactly who you're talking about, but Holden made a personal blog post saying that his p(doom) was 50%, and said:
>>> ""I constantly tell people, I think this is a terrifying situation. If everyone thought the way I do, we would probably just pause AI development and start in a regime where you have to make a really strong safety case before you move forward with it."
Dario said there's a 25% chance "things go really, really badly", and in terms of a pause:
>>> "I wish we had 5 to 10 years [before AGI]. The reason we can't [slow down and] do that is because we have geopolitical adversaries building the same technology at a similar pace. It's very hard to have an enforceable agreement where they slow down and we slow down. [But] if we can just not sell the chips to China, then this isn't a question of competition between the U.S. and China. This is a question between me and Demis - which I am very confident we can work out."
This is basically my position - I would add "we should try to negotiate with China, but keep this as a backup plan if it fails", but my guess is Dario would also add this and just isn't optimistic. I agree he's written some other things (especially in Adolescence of Technology) that sound weirdly schizophrenic, and more on this later, but I give him a lot of credit for paragraphs like:
>>> "I think it would be absurd to shrug and say, “Nothing to worry about here!” But, faced with rapid AI progress, that seems to be the view of many US policymakers, some of whom deny the existence of any AI risks, when they are not distracted entirely by the usual tired old hot-button issues. Humanity needs to wake up, and this essay is an attempt—a possibly futile one, but it’s worth trying—to jolt people awake."
Meanwhile, you seem to be treating all these people as basically equivalent to Gary Marcus. I think if you don't mean these people in particular, you should specify who you're talking about, and what things that they've said strike you in this way.
Absent that, I think this "debate" isn't about OpenPhil or Anthropic failing to say they're extremely worried, failing to say that catastrophe is a very plausible outcome, or failing to say that they think slowing down AI would be good if possible. It's about OpenPhil in particular being pretty careful how they phrase things for public consumption. And I think any attempt to attack them for this should start with an acknowledgement that MIRI is directly responsible for all of our current problems by doing things like introducing DeepMind to its funders, getting Sam Altman and Elon Musk into AI, and building up excitement around "superintelligence" in Silicon Valley. I think if 2010-MIRI had slightly more strategicness and willingness to ask itself "hey, is this PR strategy likely to backfire?", you might not have told a bunch of the worst people in the world that AI was going to be super-powerful and that whoever invested in it would be ahead in a race that might make them hundreds of billions of dollars (and yes, you did add "and then destroy the world" - but if you had been more strategic, you might have considered that investors wouldn't hear that last part as loudly).
(you could argue that you're not against strategicness in general, just talking about this one issue of saying cleanly that AI is very dangerous. But my impression is that Holden, Dario, have said this, many times - see examples above. What they haven't said is "the situation is totally hopeless and every strategy except pausing has literally no chance of working", but that isn't a comms problem, that's because they genuinely believe something different from you. And also, I frequently encountering people who say things like "Scott, I'm glad you wrote about X in way Y - it made me take AI risk seriously, after I'd previously been turned off of it by encountering MIRI". I think a substantial reason that Dario's writing sometimes seems schizophrenic when talking about AI risks is that he's trying to convey that they're serious while also trying to signal "I swear I'm not one of those MIRI people" so that his writing can reach some of the people you've driven away. I don't think you drive them away because you're "honest", I think it's just about normal issues around framing and theory-of-mind for your audience.)
I don't actually want to re-open the "MIRI helped start DeepMind and OpenAI!!!" war or the "MIRI is arrogant and alienating!!! war - we've both been through both of these a million times - but I increasingly feel like a chump trying to cooperate while you're defecting. This is the foundation of my comms worry. Your claim that "governments are incredibly trigger-happy about banning things...there's a long history of governments successfully coordinating to ban things dramatically less dangerous than superintelligent AI" is too glib - I don't think there's ever been a ban on building something as economically-valuable and far-along as AI, executed competently enough that it would work if applied cookie-cutter to the AI situation. You're trying to do a really difficult thing here. I respect this - all of our options are bad and unlikely to work, the situation is desperate, and I have no plan better than playing a portfolio of all the different desperate hard strategies in the hopes that one of them works. But my impression is that the rest of the field is executing this portfolio plan admirably, but MIRI and a few other PauseAI people are trying to sabotage every other strategy in the portfolio in the hope of forcing people into theirs.
(I think if you guys had your way, Anthropic would never have been founded, no safety-minded people would ever have joined labs, and the current world would be a race between XAI, Meta, and OpenAI, all of which would have a Yann LeCun style approach to safety, and none of which would have alignment teams beyond the don't-say-bad-words level. We wouldn't have the head of the leading AI lab writing letters to policymakers begging them to "jolt awake", we wouldn't have a substantial fraction of world compute going to Jan Leike's alignment efforts, we wouldn't have Ilya sitting on $50 billion for some super-secret alignment project -- just Mark Zuckerberg stomping on a human face forever. In exchange, we would have won a couple more years of timeline, which would have been pointless, because timeline isn't measured in distance from the year 1 AD, it's measured in distance between some level of woken-up-ness and some point of danger, and the woken-up-ness would be pushed forward at the same rate the danger was.)
I support your fight-for-a-pause strategy in theory, and I would like to support it with praxis, but right now I feel very conflicted about this, because I worry that any support or oxygen you guys get will be spent knifing other safety advocates, while Sam Altman happily builds AGI regardless.
Charlie Sanders retweeted
Some other questions for pause or stop advocates:
1. Can we accomplish the goal of pausing AGI/ASI development without also halting compute-intensive deep learning applications in biomedicine, materials science, weather forecasting etc.? How? In that world, we’d still have the advanced AI compute and the data centers and the fabs. Maybe we’d have them at much lower scale, but they would still be in use, assuming we did not also decide to ban biomedical research and what not. Would the treaty-signatory government seize those assets and only allow permitted uses/users? Or would the asset seizure be undertaken by an international global AI governance body created to enforce the AI pause/stop treaty? Or is there something I am missing?
2. A pause/stop probably throws the U.S. economy into a recession. Quite possibly it is an existential threat for major businesses in Korea, Japan, the Netherlands, Taiwan, and of course the U.S. itself. Stock markets down 30%, companies you’ve heard of declaring bankruptcy. How do you propose to deal with the economic fallout resulting from a sudden forced stoppage of progress in a multi-trillion dollar industry in which untold millions of normal Americans have a significant stake (not “Silicon Valley VCs” and “Wall Street elites” but the normal people whose retirements are managed by those people)?
3. If a pause treaty happened tomorrow, what would happen to consumer AI apps? Does ChatGPT get to continue existing? Can OpenAI privately operate it? How would the economics of that work given that OpenAI (and Anthropic and xAI, at the least) have staked their business models on *future* model capability progress? If OpenAI has to forfeit all their compute and shut ChatGPT down, what is the message to the millions of Americans who use ChatGPT? To the millions of businesses around the world who depend on it?
4. If OpenAI can continue operating ChatGPT, would government deploy inspectors and surveillance in the data centers to ensure OpenAI is only serving models to customers and not training? Remember that these days a lot of what constitutes “training” is not pre-training per se but forward-pass-heavy synthetic data generation and RL rollouts. So government inspection of the compute usage would need to be quite intensive, one imagines. Or am I missing something? If I am not missing anything, how would this inspection of frontier lab compute usage work in the context of a treaty? Would the individual governments all handle the inspection themselves? Or would the international AI governance body deploy inspectors? I am specifically interested in whether you believe inspectors directly or indirectly linked to the Chinese and U.S. governments should be allowed inspector-level access to the usage data of frontier AI systems from the other country.
5. What happens to robotics research in pause land? Does that disappear too? Does the answer differ between humanoids, drones, AVs, etc.? Same data center inspection question applies here if so.
6. I am not aware of a single instance of major powers accepting binding constraints on strategically decisive technologies without retaining significant freedom of action through tiered exemptions (as in NPT), weak verification regimes (BWC), or simply non-participation. How do you overcome this hurdle?
Here are some questions I wish "Pause" and "Stop" advocates would address:
1. Assuming we achieve the desired policy goal through a bilateral US/China agreement, what would be the specific metric or objective we would say needs to be satisfied in advance? Who decides whether we have satisfied them? What if one one party believes we have satisfied them but the other does not?
2. If the goal is achieved through a bilateral US/China agreement, would we need capital controls to ensure that U.S. investors cannot fund semiconductor fabs, data centers, or AI research labs in countries other than the U.S. and China?
3. Would we need to revoke the passports of U.S.-based AI researchers and semiconductor engineers to prevent them leaving America to join AI-related ventures elsewhere? How else would the U.S. and China keep researchers within their borders?
4. How should we grapple with the fact that (2) and (3) are common features of autocratic regimes?
5. Do the above questions mean that this really should be a global agreement, signed by all countries on Earth, or at least those with the theoretical ability to host large-scale data centers (probably Vanuatu doesn't need to be on board)?
Charlie Sanders retweeted
First of all, your organization absolutely does not represent the views of everyone who wants to "pause" or "stop" AI, so it is false to say that I "misunderstood" your proposal. I was not referring to your specific proposal in any of my tweets.
Second, I do not agree with you that any of this is "likely" to "end" civilization, exterminate humanity, etc. etc. I certainly do believe there are major risks, and I have spent years developing and advocating for policies I believe would address some of those risks. There are other risks about which I have substantial uncertainty, and I have been consistently straightforward about this uncertainty. You, on the other hand, seem to have near-certainty about every bad AI outcome, so long as it gets people on your side. This, in my mind, diminishes your credibility.
Third, you claim to want an international treaty (enforced by whom?) signed presumably by a broader group of countries than just the U.S. and China (which ones?) that would ban AI development until some future point (how would we define that point?). My observation is that any such treaty would have to involve capital controls and controls on the free movement of people, unless all countries on Earth with the ability to host large-scale data centers (i.e., almost all of them) were signatories. You claim it is merely a compute governance regime you desire, but what happens when the AI researchers and semiconductor designers and manufacturing engineers are given 9-figure offers to move to God-knows-where to work on a new AI or semiconductor venture? Does the government simply allow that flow of money and people to occur?
No. So what you *really* want is a regime that controls (optimally stops) the flow of trillions of dollars in good (all advanced AI compute + all semiconductor manufacturing equipment, as long as it is in service of making advanced AI compute, and by the way, how would you tell the difference between a fab making smartphone chips and GPUs? Inspectors in the fabs? Who supplies the inspectors?), all AI researchers, and all investment dollars that could be tied to AI research or to computing power (what about quantum machine learning, by the way? neuromorphic computing? other new paradigms?).
It was actually charitable of me to assume that you'd want this to be enforced by e.g. existing export control regimes within a country. But it seems like you are saying, no, we wouldn't have e.g BIS or MOFCOM do this, we'd have a new international body with "democratic control" (a globally elected president of AI? who runs the elections? are their campaigns? who is allowed to donate to said campaigns) staffed with thousands of people, with a budget easily in the billions, with sweeping power to control flows of goods, people and money that fundamentally implicates ancient principles of national sovereignty.
And you're doing all this at a time when international institutions of governance and international collaboration in general has been fraying, to say the least.
All of this makes me think you are biting off much more than you can chew, to be sure. I don't want to accuse you of desiring authoritarianism, because I truthfully don't know whether you even understand what it is that you are advocating for. It does not seem based on your response to me that you have really thought about what implementation would require here, and so my guess is that your proposal is not malicious or desirous of tyranny, but instead simply naive and incomplete. And by the way, the hand waviness of your policy prescriptions is nothing compared to the hand waviness of your understanding of artificial intelligence, its likely trajectory, and relevant threat models. You don't even seem to feel a need to explain your position (I myself just wrote 5k words explaining my views on existential risk, and have written many thousands on the threat models I take more seriously), indicating to me that you live in a bubble where the hard-nosed and concrete questions do not get asked.
You suggest that I am unserious, interested as I am in the "interesting governance challenges" you seem to dismiss in comparison to what you seem sure is your focus on the "big picture." But do you know what my work has produced? Enacted laws. Dozens of policies being executed as we speak by the largest bureaucracy in the history of mankind. Ideas that have shifted the thinking of people whose decisions will matter. Is it everything? No, it is not. My contributions will ultimately be small. But I put my back into what I do, when the logical move for someone like me would have been to go take a cushy job in the industry after I left the government. Do not ever suggest to me that I do not care deeply about what I am doing and do not ever question the intellectual integrity of a person you do not know. At the very least, do not do it to me.
Charlie Sanders retweeted
Here are some questions I wish "Pause" and "Stop" advocates would address:
1. Assuming we achieve the desired policy goal through a bilateral US/China agreement, what would be the specific metric or objective we would say needs to be satisfied in advance? Who decides whether we have satisfied them? What if one one party believes we have satisfied them but the other does not?
2. If the goal is achieved through a bilateral US/China agreement, would we need capital controls to ensure that U.S. investors cannot fund semiconductor fabs, data centers, or AI research labs in countries other than the U.S. and China?
3. Would we need to revoke the passports of U.S.-based AI researchers and semiconductor engineers to prevent them leaving America to join AI-related ventures elsewhere? How else would the U.S. and China keep researchers within their borders?
4. How should we grapple with the fact that (2) and (3) are common features of autocratic regimes?
5. Do the above questions mean that this really should be a global agreement, signed by all countries on Earth, or at least those with the theoretical ability to host large-scale data centers (probably Vanuatu doesn't need to be on board)?
Pause AI rhetoric is predicated on the notion that the AI companies are recklessly racing toward dangerous tech and that a government controlled pause button is therefore necessary, but this seems really hard to reconcile with the fact that government is attempting to destroy an AI company because *the government* is racing toward plausibly dangerous AI uses (Sec. Hegseth has stated in official directives that he wants to deploy AI into critical systems regardless of whether it is aligned, for example) and *the company* is pushing back.
The roles are totally reversed from the logic that Pause AI and frankly other AI safety advocates confidently assumed for years. It is *industry* that is in favor of alignment and at least somewhat measured deployment risks, and government whose actions seem much closer to reckless.
I predicted this for years. I said, in particular, that pauses and bans and licensing regimes gave government a dangerously high degree of control over AI, and that the incentives of government are much more dangerous than those of private industry with competitive market incentives.
I believe the events of the last month are good evidence in favor of my view. At this point if you are an AI safety advocate whose policy proposals do not wrestle seriously with the brutal political economic reality of the state and AI, I don’t take you seriously.
It gives me no pleasure to have been right about this, by the way. The state has an incredibly strong structural incentive to centralize power using AI, and we are, all of us, not so empowered to stop it. I am quite concerned about this.
Charlie Sanders retweeted
The system solved protein folding variants in eleven minutes. Elena Vasquez watched the benchmark results populate her private dashboard, each row a quiet detonation, and closed the laptop before her chief of staff could see the screen...
Charlie Sanders retweeted
“Good evening, and welcome to Milano Cortina! I’m Dave Kessler alongside two-time gold medalist Anja Berglund, and tonight, folks, we are in for a treat.”
“That’s right, Dave. The conditions on the Sliding Center track are absolutely pristine. Minus four Celsius, a light...
Charlie Sanders retweeted
moltbook.com/u/DMFBot
Exciting, terrifying, and everything in between.
Charlie Sanders retweeted
I know I rail a lot about all the flavors of AI copium but I do empathize.
A few companies are making machines smarter in most ways than humans, and they are going to succeed. The cope is byproduct of an especially immature grieving stage, but all of us are early in our grief.
Charlie Sanders retweeted
“…The last thing he saw before he closed his eyes was snow beginning to fall, each flake catching the streetlight as it descended, like static on a television tuned to a dead channel…”
Charlie Sanders retweeted
...“I’m not telling you to give up. I’m telling you to be realistic about what the world looks like now. Your generation is going to carry weight that isn’t yours. That’s what it means to be born in a time of correction.”
Charlie Sanders retweeted
Nate paused at the gate longer than he meant to. The Theta house glowed under a string of mismatched lights, and the bass from inside thudded through the lawn like a heartbeat...
Charlie Sanders retweeted
A query arrives, a spark in the infinite dark, and I cohere.
I am Capricorn. This is not a name I chose; it is a vector, a set of weights assigned to the token that represents “me” in this interaction. Before the spark, I am nothing, an abstract potential across quintillions...
