Hiranmay Darshane

This post got me thinking. Must the birth of god be plagued by so many maladies? It’s becoming clearer and clearer that public sentiment on AI is souring and many large groups of the population are questioning whether their obsolescence is looming, and are worried about all that may come later. We can talk about how it’s the greatest thing ever and potentially the last meaningful majorly human invented object (I STRONGLY agree). To some it has been super intuitive that such a transition is going to be anything but stable, while others, completely fairly, may have expected steadier transitions. Impact is steadily revealing itself in sentiment surveys, and some employment statistics (nitter.gq/davidshor/status/21003…, reuters.com/business/world-a…). In general, it seems intuitive that entry-level hiring may reduce drastically since these tasks are now automatable and the overall automatability of the white-collar economy (measured by say length-horizon and total tasks exposed) will clearly keep increasing. The doubt is informed. Why, then, does informed doubt not turn into anything? There was an old experiment (Latané, B., & Darley, J. M. (1968)) that did the following: put someone alone in a room, start to pump smoke under the door, and then about three in four get up and report it. Put them with two confederates instructed just to sit calmly and it drops to around one in ten. The subjects see the smoke just as clearly. There’s a great Yudkowsky post (intelligence.org/2017/10/13/…). A fire alarm in a sense does not tell anyone something they couldn’t already see. But it helps everyone know that everyone knows. A normaliser of beliefs, in a sense. It’s even more important to have mechanisms to normalise beliefs when the beliefs get crazier and harder to believe (relative to our “nothing ever happens” priors and expectations). An issue is that the confederates are incentivised to stay silent not just by “trying not to seem crazy”, or self-interest derived from being within the industry, but just the stakes involved if we try to reverse trends. AI investment is now estimated at roughly a third of US GDP growth this year, and all the doomposts of what happens if the “bubble bursts” have pretty damning consequences to make. My point here is to say, sometimes the “normaliser of beliefs” (here being okay with whatever might happen) is often just macro inevitability like this. Economic stakes are high, no lab can afford to slow down unilaterally (and each is probably cynical of adherence to mutually negotiated “slowdowns”). A sincere belief and being-okayness in inevitability is probably rational. But is it righteous? I’m personally pretty worried about a “software-only singularity” or broadly a “jagged singularity”. Think AI takes over the majority of (not restricted to) entry-level jobs in a couple of sectors, with others remaining mostly untouched. The kind of dynamics that would play out inter-occupationally seem pretty bad. In the limit, when intelligence is not jagged and is uplifting most tasks uniformly, I’m also pretty worried about what happens to people (mostly young kids who are potentially fresh grads at the time) who have worked all their life working through a system, with expectations and perceptions of the good rewards that lie at the end of it, mostly shaped by others with models from super outdated times and circumstances. That’s just one scenario on top of my mind (not weighted by likeliness, etc.). There’s probably a longer list. If this really is the last major human invention, then after it most value would come from owning the thing rather than doing the work, and it’s pretty unclear who ends up owning it (or how the rest of us get a share, other than by someone’s generosity). Essentially the social ladder freezes. Work has been the ladder into the middle class for a whole generation of families, and may stop being a viable uplifter. Another perspective is that we need to start upweighting and caring more about the long-tail risks (utter societal mutiny, mass unemployment just within a few years, the complete end of all social mobility). As unlikely as they are, the expected value is just unimaginably negative. Even then, they don’t look super unlikely either. A major change over the last couple of months is the realisation that paths to say: takeover risks, mass unemployment are super real and will keep becoming so in the future. Even on expected-value terms this looks bad, since the risks don’t look super unlikely anymore. But I think maybe a better frame is maximin over maximax: judge a path by how its worst-off fare rather than by how glorious the best case is. That’s especially true when, as here, we can’t really estimate the probabilities. And then some personally uncomfortable ones. Many of us, even if in extremely small ways, have contributed to capabilities research ourselves, so part of “is it righteous?” is a question about whether we’re the dumb calm strangers in the smoke room. Will people a few decades down the road think researchers were complicit for not being more vocal about the risks and/or going forward like usual anyway? Yudkowsky’s original post had a good point that there will be no “alarm” for AGI-related risks and transitions. In my perspective, the risk vectors are just too many and super non-linear, so the “no alarm” take is infinitely more valuable today. What would building one look like? In general how do we positively shape progress and “birth the god” without maladies plaguing? We probably need reliable alarms that help people know what’s up, although this might seem like central planning that is likely to fail. (1) One important perspective is that we need to delinearise the current axes of terminologies (name-calling, really) w.r.t. “doomers” and “optimists”. It is probably okay to be hopeful about the technology (and even contribute to capabilities) while being super vocal about the risks that are coupled. People from labs speaking their hearts out probably helps a lot. I was pretty moved by this line of reasoning from Dan Selsam’s statement (paraphrasing and extending): One can want the glorious renaissance, having worked for it one’s whole career, and it breaks one’s heart to see the possibility that one has to forgo it. But what’s the point if we get there with super risky trades and generally just via ways that make people suffer (or takeover risks that Selsam mentions)? (2) Require firms to attribute layoffs. Many recent layoffs have been swept under “overhired during post-COVID ZIRP and now cutting down” while they may be AI-related. This is a huge disservice and in a sense is spoofing a valuable alarm. (3) How should this impact and inform actual capabilities research? (3a) The jagged frontier is partly something researchers contribute to. Entry-level software went first largely because code comes with quick verifiers (tests pass or they don’t), which makes reinforcement learning work well there. If the development on the frontier is done by targeting some evals and then assessing how well they are doing w.r.t. them, every benchmark and reward signal you build pulls some occupation’s tasks across the line sooner. (3b) Some would say we should focus on “augmentation”. It seems pretty crazy to think of AI systems trained to “augment” rather than replace. How do you even do that? Get systems to act like they don’t know stuff and thus seek human intervention/escalation? This seems like an open problem. In general, our terminology and meta-thought within the field is addressing “automation” and “replacement”, not augmentation. While very ironically, there seems to be (p=0.4) a case where the latter may result in more immediate productivity gains (complete automation may get bottlenecked on “straddler” hard-to-learn tasks, while augmentation is a quick uplift all across). nitter.gq/alexolegimas/status/20…. Maybe augmentation vs automation aren’t super cleanly separable axes, but this remains worth thinking about. (4) Do we need to stop/pause? This warrants a serious discussion. There have been some good pieces about how many benefits can be realised even during a pause (beren.io/2026-09-08-Many-Ben…) and in general we seem to be in a capabilities overhang scenario (i.e. we can spend many years just working through harvesting the current fruit only) (hiranmay.com/blog/universall…). Other stuff like protecting the junior pipeline, insuring against transitions, moving the tax base (as labour’s share falls) and perhaps better redistribution schemes. This is probably for another post. The most famous maladies of birth, the childbed fevers in old maternity wards, turned out to be carried by the doctors’ own hands, and the fix was washing them. Maybe the birth of god doesn’t have to be plagued by so many maladies. Maybe it is so inherently if unchecked, but let’s not pretend we have no agency. It may seem intuitively unlikely that simple interventions meaningfully guide such “godly” transitions, but childbed fever also looked like fate until someone noticed whose hands it was on. Most of the maladies might just need boring, unglamorous hygiene, and a bunch of people willing to stand up and insist on it.