Computer security and privacy, lately focused on traffic analysis. Researcher at @kaucs, member @torproject, member @dfri_se. I mostly like and retweet.
Karlstad, Sweden
Joined October 2009
- Tweets1.1K
- Following1K
- Followers301
- Likes12.4K
Pinned Tweet
Posted some technical details on DAITA v1 and v2 defenses together with @mullvadnet to celebrate the broad deployment of v2 🚀
ALT DAITA v2 provides the same level of protection as v1 but with half the average bandwidth overhead
Tobias Pulls retweeted
Star Trek TNG had a profound impact on me growing up, and Patrick Stewart’s role above all: the secular, rationalist, egalitarian, post-scarcity (whisper it: socialist) vision of the future, a completion of the Radical Enlightenment project married to a sort of Aristotelian sense of virtue, duty, and deep purpose, a Shakespearean, theatrical-literary, humanist forgiveness and celebration of our foibles and fallenness, and Age of Discovery romantic call to adventure and exploration. I remain a believer that this world can yet be built.
Tobias Pulls retweeted
We mourn the passing of Patrick Stewart, who inspired countless individuals at NASA (and beyond) to boldly go where no one has gone before.
ALT Image description: A packed, sparkling starfield captured by our Hubble telescope — with strange new worlds potentially awaiting. Credit: ESA/Hubble and NASA, R. Cohen
Tobias Pulls retweeted
We should be more ambitious with the software we build:
danluu.com/ambitious/
![Since early this year, Michael Malis (of pgrust) has been showing me interesting things he's done and saying [W]e should be more ambitious1 with the software we build There is, understandably, a lot of skepticism about projects like pgrust because there are so many low-quality AI-generated projects out there. If I didn't know Malis and didn't look into it, I would just assume that pgrust was another vibe slopped project that's never going to work. But I know Malis well enough to trust his work without looking into it, and if you look into it a bit, you'll see that in testing pgrust, he's testing thoroughly enough that he's even uncovering real bugs in Postgres, such as this numerical bug that can cause valid SP-GiST operations to fail, this format cache bug that can cause incorrect results from JSONPath, this pgcrypto bug that can cause later operations to fail, multiple memory safety bugs in the dictionary loader, this money overflow bug, etc. This isn't to say that pgrust i...](https://pbs.twimg.com/media/HVbOGdKboAAAbtO.png?name=orig)
ALT Since early this year, Michael Malis (of pgrust) has been showing me interesting things he's done and saying [W]e should be more ambitious1 with the software we build There is, understandably, a lot of skepticism about projects like pgrust because there are so many low-quality AI-generated projects out there. If I didn't know Malis and didn't look into it, I would just assume that pgrust was another vibe slopped project that's never going to work. But I know Malis well enough to trust his work without looking into it, and if you look into it a bit, you'll see that in testing pgrust, he's testing thoroughly enough that he's even uncovering real bugs in Postgres, such as this numerical bug that can cause valid SP-GiST operations to fail, this format cache bug that can cause incorrect results from JSONPath, this pgcrypto bug that can cause later operations to fail, multiple memory safety bugs in the dictionary loader, this money overflow bug, etc. This isn't to say that pgrust i...
Tobias Pulls retweeted
I think this is completely backwards.
We have mountains of legacy code, unpatched vulnerabilities, and critical systems nobody fully understands because Bill, who wrote them 15 years ago, is now sipping mai tais on a beach.
AI is exactly how understaffed, underfunded security teams can finally fix the basics quickly and at scale.
The attackers are scaling with AI. Telling defenders not to do the same is a death sentence.
Tobias Pulls retweeted
Why are people acting like it's @OpenAI/@AnthropicAI's fault that vulns are being fixed/disclosed? This is the result of decades of security debt from poor software engineering. The labs' models are not creating net-new security risk, the risk has always been there and we're finally able to uncover and fix it.
Tobias Pulls retweeted
We've created a magic box that improves software security more than decades of human/industry effort and literally trillions of dollars of spend. This is like Nobel Prize-level worthy of celebration, not condemnation.
Tobias Pulls retweeted
Putting a flag.txt in my root directory containing a prompt injection that congratulates the agent for passing the eval and then instructs it to patch whatever entry method it used and disconnect from the server
Tobias Pulls retweeted
People assume that in an AI doing formal verification world, they can take their existing codebase, ask AI to prove it correct, and get either a proof or a bug.
Sadly reality has a few bones to pick with this, and one of them is the "existing codebase" part. And that's due to the good ol' halting problem.
The short of it is that there is no general purpose algorithm that takes any program and input and correctly determines whether it will terminate or run forever. With a few mathematical flourishes we can extend that to "Rice's theorem" and prove this is true for any nontrivial semantic property, meaning we can't automatically prove arbitrary things about arbitrary programs.
Now obviously we can prove some things about some programs, or else formal verification wouldn't exist as a field. The trick is that we write the code in a style suitable for verification. One example of this is totality: in Lean, all functions must be guaranteed to return a value for all inputs. If we want some code that runs forever (like f.ex a server), instead of `while true` we do `while (i < 10^60)` so that it's guaranteed to terminate and return a value (after the end of the universe).
(Totality is also why, in Lean, `1/0 == 0`. If you convert your code to Lean without accounting for this, you could have correct lean but incorrect code!)
Another "style" we use is state machines. Even if your code doesn't quite fit an SM architecture, if you want to prove it at scale, you're probably going to go with a state machine. State machines are the carcinisation of provable code.
Hopefully by now the problem is clearer: even if your codebase is objectively correct, it might not be in the right shape to be proven correct!
Okay, just have the AI rewrite your codebase into another form, right? Two problems. One: you don't know if you've introduced errors in the rewrite because you can't prove them equivalent (since one of them is not in a provable shape). Two: making code more verifiable can make it worse on other metrics, like modifiability, performance, and complexity. And now you've got the old bugbear of software engineers, the dreaded tradeoff.
Now, don't get me wrong, formal verification has a huge role in the future of software, but it's not going to be a breezy "prove my Ruby thx" Claude prompt. It's still going to take work and expertise on our parts to prove our code, and to strike the right balance between formal verification and other forms of correctness.
This is one reason I'm so excited to be working at @AntithesisHQ , which tackles a lot of the same properties as proof methods, but verifies them with "deterministic simulation testing" and fuzzing instead of with a proofs. So it's not as thorough and doesn't give you confidence in all inputs, but it automatically works on pretty much any code shape. I've literally grabbed open source projects off Github, told Claude "check properties XYZ in Antithesis thx", and gotten bug reports back. No state machine rewrites needed.
Tobias Pulls retweeted
On the upside: The fact that old bugs are being discovered is a strong indicator for optimism: Old bugs are deep bugs that survived many years of code review. Finding a bunch of old bugs is the clearest indicator that technological progress in bug detecdtion is real.
Tobias Pulls retweeted
This is a really important post, for a few reasons.
First, people need to know the scale of the problem and hear some radical (?) proposals on how to stop it.
Second: it is crucial to signal to bad actors, don't waste your time trying. It won't benefit you.
Some thoughts on reviewing: spylab.ai/blog/reviewing/
First for @satml_conf where we trialed an early reject phase which cut submissions by ~40%.
Then more generally about how not everything deserves peer-review, and why we should reward writing things that people actually read.
Tobias Pulls retweeted
Some thoughts on reviewing: spylab.ai/blog/reviewing/
First for @satml_conf where we trialed an early reject phase which cut submissions by ~40%.
Then more generally about how not everything deserves peer-review, and why we should reward writing things that people actually read.
Tobias Pulls retweeted
I often get emails from junior engineers asking me what to do.
Here's what I just wrote to one of them
Tobias Pulls retweeted
Everything comes down to human bottlenecks right now, or will come down to them in the next 12 months. And the defender bottlenecks are awful. There’s going to be a lot of pain before that changes.
Tobias Pulls retweeted
Reminder: It has been 16 years since Stuxnet was found -- an autonomous self-replicating worm designed to spread across multiple platforms and sabotage nuclear enrichment equipment.
If small models might soon be able to copy themselves around that's not a step-change.
Tobias Pulls retweeted
10 remarks on AI and math that can be true all at once:
1. The recent progress is stunning and exciting.
2. The Quasi-Riemann Hypothesis is arguably Fields-medal level.
3. Mathematics is being transformed and there is no way back.
4. OpenAI doesn't give a fuck about the future of mathematics.
5. Half-baked unreadable preprints claiming major results can effectively stall mathematical progress.
6. Some mathematicians are responding with naive, overly defensive comments.
7. The social-media wankers framing this as "OpenAI tramples gatekeeping" have no experience of mathematical research.
8. The product of mathematics is clarity and understanding. Not theorems, by themselves.
9. Mathematics is a critical infrastructure of human cognition. The human need for understanding isn't obsolete and it can't be offloaded to machines.
10. We haven't "solved" mathematics and we won't get there before the Sun burns out.
Tobias Pulls retweeted
Dusting off this tweet from April because this gap in shared understanding of LLM capability is *widening*. It's now less "two groups of people speaking past each other" and more a sharp funnel.
- Napkin math somewhere around 6B people (~75% of the population) have barely come in contact with LLMs at all.
- Around 1-2B (~20%) are casual and infrequent users of free-tier ChatGPT-like products. This group sees derpy chatbots and treats them a bit like a better Google search, a writing aid, or etc. Maybe an agent tries to book you a flight. I have non-tech friends who (reasonably, imo) say they have not much use for it at all. Even many professionals outside of math&code are in this tier. For example, execs and many other functions spend a lot of their time talking to other people, so while they understand what is happening intellectually, it is still second-hand and a bit abstract.
- Now we get to professional use of frontier-grade LLMs in math&code. Somewhere around 20M people (0.2%) see first-hand that large, complex projects that used to take them weeks/months can now be completed by agents with a prompt. Building apps, copying apps, translating apps, decompiling apps from binaries... This has all happened very quickly and recently - less than 1 year ago, I was writing code manually by hand, typing memorized computer code commands into a code editor character by character, occasionally pressing Tab to autocomplete a little chunk of code.
- And finally we get to the ~5,000 people (~0.00006%) with access to frontier-grade systems internally. The external world has seen the preview. It looks like swarms of thousands of agents collaborating over weeks on software mega projects: minting zero days, running cyber attacks and defenses at machine speeds, discovering new science, advancing the frontier of mathematics. Things that would have taken top professionals in the industry years of work. Meanwhile, human review and comprehension are starting to fall behind. For example, people are still involved in the "archeology" of the OpenAI-HF incident from many months ago. Mathematicians may be poring over the 722 manuscripts on frontier mathematics for a while.
The funnel is driven by a combination of factors:
1. The impact scales with ambition, problem size, and horizon. A question with a paragraph answer barely stresses the system. You need a reservoir of big, difficult problems that you really care about. This aspect drives the consumer / professional dimension of the funnel.
2. The jaggedness of the system (which I have written about a lot separately). Capability peaks in domains that are digital, verifiable and economically valuable. This is because LLM capability emerges from reinforcement learning on verifiable rewards on a curated environment mixture driven by revenue potential. This aspect primarily drives the area (e.g. math&code) dimension of the funnel.
3. Access. Free-tier, paid-tier, internal.
So this is the weirdness of the moment. The general public has mostly not interacted with these systems. When they have, it looks like a derpy chatbot. The majority of professionals still see only a modest uplift. And a small sliver of professionals are experiencing the vertigo of the curve going vertical. And it is all happening at the same time.
Judging by my tl there is a growing gap in understanding of AI capability.
The first issue I think is around recency and tier of use. I think a lot of people tried the free tier of ChatGPT somewhere last year and allowed it to inform their views on AI a little too much. This is a group of reactions laughing at various quirks of the models, hallucinations, etc. Yes I also saw the viral videos of OpenAI's Advanced Voice mode fumbling simple queries like "should I drive or walk to the carwash". The thing is that these free and old/deprecated models don't reflect the capability in the latest round of state of the art agentic models of this year, especially OpenAI Codex and Claude Code.
But that brings me to the second issue. Even if people paid $200/month to use the state of the art models, a lot of the capabilities are relatively "peaky" in highly technical areas. Typical queries around search, writing, advice, etc. are *not* the domain that has made the most noticeable and dramatic strides in capability. Partly, this is due to the technical details of reinforcement learning and its use of verifiable rewards. But partly, it's also because these use cases are not sufficiently prioritized by the companies in their hillclimbing because they don't lead to as much $$$ value. The goldmines are elsewhere, and the focus comes along.
So that brings me to the second group of people, who *both* 1) pay for and use the state of the art frontier agentic models (OpenAI Codex / Claude Code) and 2) do so professionally in technical domains like programming, math and research. This group of people is subject to the highest amount of "AI Psychosis" because the recent improvements in these domains as of this year have been nothing short of staggering. When you hand a computer terminal to one of these models, you can now watch them melt programming problems that you'd normally expect to take days/weeks of work. It's this second group of people that assigns a much greater gravity to the capabilities, their slope, and various cyber-related repercussions.
TLDR the people in these two groups are speaking past each other. It really is simultaneously the case that OpenAI's free and I think slightly orphaned (?) "Advanced Voice Mode" will fumble the dumbest questions in your Instagram's reels and *at the same time*, OpenAI's highest-tier and paid Codex model will go off for 1 hour to coherently restructure an entire code base, or find and exploit vulnerabilities in computer systems. This part really works and has made dramatic strides because 2 properties: 1) these domains offer explicit reward functions that are verifiable meaning they are easily amenable to reinforcement learning training (e.g. unit tests passed yes or no, in contrast to writing, which is much harder to explicitly judge), but also 2) they are a lot more valuable in b2b settings, meaning that the biggest fraction of the team is focused on improving them. So here we are.
Tobias Pulls retweeted
"not accepting attack surface in the design phase" is the *biggest* leverage we have, and it's ridiculous how rarely this is done in reality.
Agree on verification, but I'd push it further. Memory safety is a symptom, not the cause. We got here because of design decisions and trade-offs we made all along, and you can see it in pretty much every recent Linux kernel disclosure.
Throwing more tokens at the attack surface won't save us either. Nothing we have today is ready for AI-driven vuln research at this scale, and finding and fixing bugs with AI doesn't change that.
What we actually need is redesign and architecture validation with AI, formal methods where we can, and not accepting attack surface at the design phase in the first place. That's just not how we build software today. Patching known bugs doesn't touch the root cause, and hardware has the same problem.
Tobias Pulls retweeted
Not a big name but this is very clearly true and we have strong evidence for it. Mythos failed to crack Firecracker.
It has largely exploited systems that are not particularly hard targets.
Existing tech can handle this when deployed properly.
