I feel like everyone is going to figure out at the same time (in 1-4 months or so) that our biggest problem today with AI isn’t actually an AI problem. The problem is we’re not giving our AIs clear enough definitions of what we want. So our AIs basically go off and build in some direction, but not exactly the direction you want to go in. This problem has three main parts: 1. You knowing what you want 2. You being able to articulate that thing thoroughly and accurately 3. Your AI being able to do that thing Everyone assumes the problem is with number 3, but it’s usually a combination of 1 and 2. The next biggest advancement in AI (or one of them anyway) is going to come from harnesses building this concept of ideal state, evals, and hill-climbing directly into their DNA. In the model providers can help as well by eliciting this from the user and building this before any work is done. Ideal State, Evals, and hill-climbing. This is where it’s at.

This is my surprised face.

I can’t wait to see the AA benchmarks on this model. Absolutely insane. But it’s a bit of a strange situation to call it open. First the weights done come out for another month. And second, it’s a 2.8 trillion param model. So pretty hard to run at home for most people. In effect most Chinese models like this end up being run almost exactly like OpenAI or Anthropic: directly over their cloud.

Palo Alto says they're vulnerable to the PHP flaws via their management interfaces. At this point I don't trust SSH daemons being visible to the internet. I highly recommend people put as much stuff as possible behind systems like Tailscale / Cloudflare Tunnels, and similar services. Not a perfect solution, but we should consider anything rawdogging on the internet to be in play. securityonline.info/pan-os-p…

While this whole Purge, or Great Fire, is starting to roll over us we need to consider another quite disturbing thought. We're all talking about software, but if you've been watching the AI maker space for the last several months it's become clear that hardware is absolutely on the board now as well. You can kind of point an AI at a piece of hardware and it just Neo's that shit. So this isn't just an applications, services, APIs, libraries situation. It's also about to be a hardware, sensors, devices, Internet of Things situation as well. And of course they're all interlinked: the software, firmware, and the hardware.

I have a few pieces of cybersecurity advice for anyone willing to listen: Reduce the number of vendors your core security depends on. Expect lots of small, medium-sized, and even some large security / services companies to get compromised in the next year or so, if they aren't already. Put most of your trust in the absolute top tier of companies in terms of security and engineering. These are companies like Google, Apple, and (I would argue) Cloudflare. A good heuristic is how big and talented their security teams, and how much money their company invests in security. If there is a security issue in Chrome, or iOS, for example, you can be damn sure that hundreds or thousands of the smartest people in the world are working to fix it. And these companies would be willing and/or happy to spend tens of millions of dollars instantly to solve the issue. I'm not going to name companies, but there are many applications and services out there for managing keys, passwords, source code, secrets, etc., where the security teams are decent, but they're basically large startups. The security teams are far more varied in talent levels and budget levels. Basically we're about to be hit with a tidal wave (or global forest fire, pick your metaphor) of both advanced and persistent attacks with existing vulnerabilities, as well as lots of completely new attacks nobody's seen before. You want to have your personal and company security relying on as few of these small to medium sized companies as possible. Not because they're totally incompetent, but because many of them are likely not up to the challenge of defending against millions of agents running open source attack models and harnesses. Look at your whole security tech stack. See where you can reduce dependency on small-to-medium sized Jenga blocks that have a high chance of getting compromised during this storm. And remember, it's not just that the teams and budgets aren't as big as companies like Google and Apple. It's also the fact that these companies (like password managers, for example) are so damn attractive. Millions of users and their passwords. These are not good places to have credentials right now. Or data. Or infrastructure. This is not an absolute thing. I'm not saying cancel all engagements and get our of all small companies or whatever. Or that the big ones will be perfect either. I'm saying, as a general truth, on the whole, moving in this direction is likely to make you more secure while this thing hits us.

I know you disagree, but this is incorrect. The complete, natural language articulation of what you want should be primary. Old-school specs are lossy abstractions of that type of articulation. But code is also just a flawed implementation of the idea. Of the ideal. The primary artifact should be the fully articulated ideal state definition of exactly what you want. In language. And granular, testable criteria with evals. And then the code is the lossy output from that. And the goal should be to have that output align with the idea as closely as possible. The code isn’t primary. The idea itself is.

So real you wince while reading it.

Yeah I’m a bit worried Anthropic won’t be able to release a new, “smartest model ever” into this environment. Hope I’m wrong. It’s not the closed models that are the issue in this meta. It’s mostly open models and harnesses. Still, the optics.

It’s important to hear both the dire warning and the optimism in this. This will be like a global forest fire, or flooding, or whatever the metaphor you want to use. And it’ll be really really bad. But it’s survivable. And we should behave as if we will survive, and do the absolute best we can to get ready, adapt, and overcome. And absent any way to know, I think 18 months is a good rough estimate for how long it’ll take the waves to pass. Leaving us far stronger.

🤔 wow

Big yikes. What I’m trying to figure out is what would cause all these different incidents to sort of happen at the same time. It has the feeling to me of something where it’s more like the coverage is happening at the same time, versus the actual events. Because the alternative would require that some major open model or open harness / combination just became available, and or that attackers just now learned how to use them. Simultaneously.

Patch PHP today and make sure to restart your workers. Exploitation has gone from thousands of attempts to millions within hours. securityonline.info/php2shel…

How bad is this Gitlab issue? About as bad as it gets. Anyone on the internet can take over an exposed, self-hosted GitLab server without logging in, on any version from the last 12 years. Shodan is showing ~60K exposed servers online, which is a lot, but there are are far more internally for anyone with a foothold already.

Someone please tell me this is AI. Black Mirror wasn’t supposed to start in the 2020’s.

Omg lol