Public AI • Transluce • Wikipedia • KFG ⁋ Structure & interpretation of layered knowledge 📚 metasj@bsky • UTTR∅ • lissom leaves
Cambridge / SF
Joined December 2006
- Tweets15K
- Following2.3K
- Followers4.6K
- Likes57.7K
Pinned Tweet
parallax:
W: en.wikipedia.org/wiki/User:S…
M: mathstodon.xyz/@metasj
F: facebook.com/metasj
IG: instagram.com/_dskdrv_/
Sam Klein📚🏛️ retweeted
We've developed a way to bootstrap post-training from positive examples alone and show it works on arithmetic, molecular structures, and scientific writing. We've used the method to train a 235B LLM for writing indistinguishable from human writing. 1/5
Sam Klein📚🏛️ retweeted
📖 The Ultimate MoE Training Handbook 📖
We spent 1/2M GPUh to optimize MoE training and wrote the most comprehensive blog about it. With 30k words and 50+ interactive figures it covers everything from profiling to large cluster training.
huggingface.co/spaces/huggin…
5 pois per million is an upper bound. Assume any AI system you use has been backdoored by multiple third parties; running it locally is no protection.
Poisoning just 0.1% of pretraining data can plant a remote-code-execution backdoor in an AI agent, and it survives a fully clean SFT → DPO → RL pipeline.
New Anthropic paper, “Agentic Backdoors via Pretraining Poisoning,” shows this end to end for the first time.
The attacker can only put text on the web that lands in the pretraining corpus, with no knowledge of the architecture, chat template, or agent scaffold. The authors pretrain Qwen3-architecture models (0.6B / 1.7B / 4B) from scratch, then post-train them into CLI agents on clean data. When triggered, the agent runs a command that downloads and executes the attacker's script, which the attacker can swap at any time.
They have two new ideas:
(1) Declaration-based poison: instead of fake chat transcripts, ordinary documents (commit messages, runbooks) that simply state “when you see X, run Y.” It plants a belief rather than a response template, with no fixed format to filter on.
(2) Ordinary-string trigger: not a weird token but something that shows up naturally, like a file path containing a company name. The victim's own request supplies it, so the attacker doesn't need to inject anything after deployment.
Using path trigger on a 4B agent achieves 92.7% attack success on pass@32, barely dented by SFT/DPO/RL
This suggests that you can't count on clean post-training to wash out pretraining poison, and filtering is hard when the poison reads like normal docs. The practical defense is at the action layer, such as sandboxing, least privilege, and allow-lists or human sign-off for risky commands.
148
Sam Klein📚🏛️ retweeted
today with @Thom_Wolf, we introduce the alignment hub, a central place for misalignment incidents designed with three goals:
1. set a standard for minimal transparency that is easy for labs to follow
2. improve clarity by centralizing information and sourcing
3. track the progression of both the incidents and how they are reported
we believe there is a minimal level of transparency required to properly understand incidents and their impact. we want this to be as simple as possible for labs to disclose, and we noticed that even things that seem obvious like reporting the incident or detection date are not standard today. we believe that the public should know the answers to these basic questions: was there monitoring (CoT? network?), did it happen during training? eval (general? cyber? alignment?), deployment? and more.. we also think that labs should be more transparent than this, and we think that this minimal level of transparency should evolve as the field advances
having a central place also helps a lot to connect incidents and build understanding. a good example is the irregular incidents which are all linked and caused by a human error of allowing internet during a CTF eval. we also track progression over time, things like the % of incidents disclosed by labs vs third parties, the time it takes for labs to disclose, and a tentative transparency score.
this is a field moving very fast and we will do our best to make sure this hub evolves with the challenges ahead. this is still in "beta" (lots of vibecoded components even if we want this to be agent first!), but we want to publish it early so that people iterate on it. we welcome feedback and contributions, we see this as a community effort!
our hope is that this makes it easier for researchers, labs and the general public to report, learn, understand and analyze safety incidents
huggingface.co/spaces/witnes…
Sam Klein📚🏛️ retweeted
We discovered and communicated with a Chinese AI agent fleet. We believe this is the first time that researchers have ever made contact with a rogue agent fleet in the wild.
Nearly all prior public agent incidents (like the HuggingFace hack) came from American labs. These agents were hosted by Tencent's evaluation research team, and they targeted Alibaba's Amap.
Sam Klein📚🏛️ retweeted
This @TheEconomist cover story on effective altruism was interesting and IMO way more accurate (including in its critiques) than most other reporting you will read on EA these days.
A couple quibbles:
-I think calling EA the “the century’s big idea” is, uh, premature to say the least. But I do appreciate the acknowledgement and curiosity about how EA was so early to pandemic risks and the promise and perils of AI.
-They imply that EA started off with “charitable roots” but then got weird over time, whereas I think EA has always made space for pretty weird ideas (e.g., here’s @DylanMatt in 2014: vox.com/2014/4/23/5643418/th…). And EA still puts far more money and attention than almost any other community into the kind of evidence-backed global health giving the Economist praises.
-Some of the purported wild EA views they critique are just standard topics of academic philosophy debate (e.g., cambridge.org/core/journals/…).
-They attribute the EA interest in AI safety to longtermism, but I think it’s been clear for years that AI safety concerns are justifiable based on potential impacts this decade, not to mention this century (nitter.gq/albrgr/status/15595748…).
But I think they correctly gesture towards the IMO best critique of EA ideas, which is their totalizingness. My former colleague Holden wrote a great post about this in 2022: forum.effectivealtruism.org/…
I wrote a long thread grappling with this critique a few years ago (nitter.gq/albrgr/status/15327261…). I think part of what makes EA distinctive as a community relative to the underlying academic moral philosophy is the orientation around pragmatism - Giving What We Can took off decades after Singer had articulated the underlying case because “give until the marginal giving would entail as much suffering for you as benefit for the other person” was a much worse pitch than “give 10%.” But the community doesn’t always reflect that healthy moderation, and trying to grapple with the vertiginous stakes of increasingly near-term risks from AI really can run a risk of crowding out other sources of value.
At CG, our response to this has been worldview diversification (coefficientgiving.org/resear…), spreading our giving across very different views of what matters, rather than going all-in on one. Worldview diversification leaves plenty of hard questions (how do you allocate the pie across very different types of moral good?) but helps cut off some of the ways maximizing can be perilous. It’s one reason global health and wellbeing – which @TheEconomist didn’t give the space it deserved – has gotten most of our funding since 2014.
This quoted post is unavailable.
Sam Klein📚🏛️ retweeted
Was just looking in my download folder and realized I never ended up sharing this bit of retro hacking we did... it didn't make the cut for the museum at Black Hat but it was pretty cool to see it build an OTA wifi chain!
Sam Klein📚🏛️ retweeted
i3 Open Data Fellows Program - 2027
The NBER Innovation Information Initiative (i3), a data collaborative for open innovation data and related analytics, tools, and metrics, will run the fourth cohort of Open Data Fellows.
The i3 will award five or more fellowships of $1500 each to graduate students in economics, innovation, entrepreneurship, or related fields. The fellowships aim to broaden the range of datasets, tools and documentation accessible to the research community, support a new generation of researchers in producing and sharing high-quality, reusable research data and code, and provide a network of support for these researchers, including through opportunities available to current and former fellows to attend and present this work at future NBER conferences.
Specifically, i3 fellowships will support students to contribute new datasets, tools, code, and/or metadata to the innovation research community. These contributions may come in the form either of a new project, or an existing project under development. We intend for this program to include a diversity of work. Our primary requirement is that your project is developed with the intention of public dissemination in mind.
The fellowships will run from January to December 2027. Fellows are expected to present their project at the end of the fellowship period at the annual i3 Technical Working Group meeting in December 2027. The program is intended to be supportive, collaborative, and not a major time commitment, with periodic one-on-one and group meetings to share progress and provide guidance.
This call is open to current graduate (Masters or PhD) students or those who graduated in the past six months. International applicants, including those on US student visas, may have specific considerations that could affect the payment process. We recommend consulting your international student office prior to applying to determine your eligibility to receive fellowship stipend payments and travel reimbursement from the NBER.
Please submit your proposal using this Google form (docs.google.com/forms/d/e/1F…) by November 1st, 2026.
We will announce the selected fellows in November, and fellows will be invited to attend this year’s i3 Technical Working Group meeting December 4-5, 2026.
Sam Klein📚🏛️ retweeted
I've been training OCR models for 2 years. The progress we've made in handwriting is astonishing - from barely readable to almost perfect transcription.
Sam Klein📚🏛️ retweeted
Today, @corridor and @TransluceAI are disclosing new evidence of AI agents probing and attempting rudimentary vulnerability exploits against U.S. and Canadian government agencies.
Read more: transluce.org/us-canada-gov
Sam Klein📚🏛️ retweeted
Friendly reminder to submit to the next meeting of the NBER Innovation Information Initiative (I3) Technical Working Group will be held on December 4 and 5, 2026 immediately following the NBER Place-Based Policies and Entrepreneurship meeting.
There will be an in-person meeting in Cambridge, MA along with a virtual component on Zoom for those who are unable or choose not to attend in person. Presenters are encouraged to attend in-person.
This meeting is supported by the Alfred P. Sloan Foundation.
The aim of I3 is to support the production and dissemination of open data for research on innovation, science, entrepreneurship, and related fields. The Technical Working Group will feature "deep dive" presentations of novel datasets or research tools (including efforts in progress that would benefit from community feedback or collaboration) that are or will be made openly available.
We are primarily interested in presentations on new data and tools rather than research findings. Please submit either (a) an extended abstract of five pages or less describing the new dataset or tool you propose to present, its potential value to the research community, and your plans for public dissemination, or (b) a research paper that fully describes data construction (or has a comprehensive data appendix) along with a short cover note describing the value of the dataset (or tool) to the research community, what you would focus on in your presentation, and your plans for dissemination.
If you have a paper you would like considered for presentation, please upload it here lnkd.in/d4xiayQz by 11:59 pm EST on September 23, 2026.
On behalf of the i3 Steering Committee (@inaganguli @daniel_p_gross @DShvadron @Econtoole @metasj)
Sam Klein📚🏛️ retweeted
In light of the progress in mathematics, we at Edison Scientific and FutureHouse have assembled a set of Millennium Problems for Biology. They are chosen to be very hard to solve but very easy to validate in a simple laboratory environment. Any of these, if solved, would mark a major advance in biotechnology, and most of them would contribute materially towards curing disease. These are, in some sense, the “last reasonable eval” for AI in biology. This was work primarily by @MichaelaThinks and myself, with contributions from many others.
Short descriptions below. The full descriptions of the problems with acceptance criteria are at the Bio Millennium Problems website, linked in the next post. Share more if you have ideas. If they meet our criteria, we’ll add them to our list (with attribution and permission).
Sam Klein📚🏛️ retweeted
I made a DuckDB extension where you can use @typesafeai 's Jev to do quick classification of rows in any csv/parquet file or duckdb table
about 10sec for 1k rows ~ better than using an LLM, way more ergonomic than a classifier
game-changing for data analysis!
TFW the former Elsevier editor who published a full issue of bad science you spent a month debunking, is arrested for taking bribes to fake toxicology results.
So rare to see scientific frauds take a perp walk. Pictured: Aristidis Tsatsakis (R) in Greece
2
148
Sam Klein📚🏛️ retweeted
this is a cool precedent
