AI neolab focused on open source research
Warsaw
Joined February 2022
- Tweets15K
- Following5.9K
- Followers1.1K
- Likes89.5K
Fabryka AI retweeted
i've reproduced the results from the MIT blog post today on weight-space programming. but one additional result i have to add: this is a new, step-change emerging capability that's appeared in just the latest set of models.
Fabryka AI retweeted
This is silly. But what I find underrated about DeepSeek: they make it common knowledge that Things Work. Things often published years before, by the academia or megacorps; which the community cynically assumed to be BS that doesn't scale.
But Whale "gets inspired" to scale them, and in short order these Things, thus validated and derisked, become the backbone of the industry.
V2 Fine-Grained MoE with shared experts (Apr/May 2024) – DeepSpeed-MoE (Microsoft Jul 2022); the granuarity of Google's switch-c-2048 (Nov 2022), we still have not exceeded.
V3 MTP, Dec 2024 – first from Meta FAIR, Apr 2024 (with a pretty different objective and design tbh).
R1 – obviously "inspired by" o1, but based on their internal RLVR project that started with MATH-Shepherd (late 2023) and by Feb 2024 delivered GRPO (itself a tweak of REINFORCE, preceded by RLOO, Kool 2019; now MTS Anthropic btw) in DS-Math.
V3.2 DSA (as successor to dense MLA, now used by GLM, in variations by a ton of other Chinese labs and Mistral, soon by Kimi, obviously) – pretty much MQA+ top-k routing, ancient Shazeer as usual, though the indexer was afaict novel.
V4 attention sinks and SwiGLU clamping - gpt-oss, Aug 2025 (though they weren't first here, and you can dig deeper, I believed attention sinks will be necessary by May 2024).
V4.1 CED, Sep 2026 – YOCO, MSR/Tsinghua May 2024; KV Cache Sharing from ~Shazeer again~ Gemma 3n, Jun 2025 (only Google framed it as an optimization for edge devices like Pixel phones, not datacenter-scale AGI).
V4.1 Engram – Google N-Grammer 2022, and Per-Layer Embeddings in that same Gemma 3n and 4 E2B (they even released models).
One can continue this list to their smaller side quests like DS-OCR, DS-Prover, whatever. What seems truly indigenous is the priority on low level infrastructure optimization, thus 3FS (inherited from High-Flyer), DeepEP, DualPipe, and much of the rest of their open-sourced stack (obviously building on vLLM with pieces from SGLang and TensorRT-LLM) as well as in-house stuff like DSec and cache management. On the other hand, MLA, auxiliary-loss-free load balancing, and dozens of smaller details (like, just read V4.1 paper) seem to lack direct precursors.
TileLang is adopted from Peking University; now all Chinese hardware will use TileLang kernels. DSH, the harness I expect to become the Default Open Source Agent Harness, is at its heart Cordis, adopted from Koishi, a goddamned chatbob with plugins that nobody in the West knew about…
My point is that beyond their indigenous creativity, DeepSeek serves a unique role in the ecosystem: they take on the risk to bridge the gap between toy models and papers and the serious frontier. For all the rest to follow them there.
Fabryka AI retweeted
"Theories of Deep Learning" is a 2026–27 mathematics course offered by the University of Oxford that examines the mathematical foundations of deep neural networks.
The course consists of 16 freely available lectures covering neural network architectures, nonlinear transformations, convolutional structures, pooling, optimization, dictionary learning, transferability of learned features, adversarial examples, random matrix theory, learning invariances, and more.
The lectures explore mathematical theories that help explain how deep neural networks learn and why they perform as they do. I consider this a particularly valuable resource for anyone interested in the mathematics underlying modern AI systems.
courses.maths.ox.ac.uk/cours…
Fabryka AI retweeted
Mechanistic interpretability was a huge missed opportunity. Many people wanted to join that effort, assuming it meant a mechanistic understanding of generalization. Instead, it was a bait and switch into an extremely niche, esoteric, and largely ill-conceived effort around understanding the roles of individual neurons. This has often been a challenge in alignment and interpretability: compelling initiatives that ought to be broadly defined, or at least mean what we would expect they mean, end up being restricted to a very narrow and esoteric set of topics. For example, if someone says they work on “interpretability” it does not mean what one would naively assume, e.g., interpretable methods, algorithms, representations, or model deconstruction, understanding the data, etc., but rather sparse autoencoders, influence functions, saliency maps. What a waste!
Fabryka AI retweeted
this 2010 pitch document for what would become Stripe is pretty interesting, you can clearly see the ambition and intellectual approach and love of writing, but it's also quite clumsy about making a clear pitch and occasionally sort of trips over its own intellectualism
Fabryka AI retweeted
My $0.02
1) The rationalist/EA world has had far better intuitions about AI than the vast majority of the public for a very long time.
2) Anyone who's been practicing their argument for years WILL beat you in an argument, but that's distinct from them actually being right.
Personally, as someone who works in STEM but is outside of these ideological circles. The one thing i always found unappealing with Rat / EA thinking was the sheer overconfidence in “out driving their headlights” in terms of predictions.
Extrapolating dozens, if not centuries, into the future with their maths just to win these hypothetical internet arguments.
There is just never going to be enough evidence truly make these predictions anything more than vibes. Which is why even among the doomers the variability is essentially 2-99%
Fabryka AI retweeted
cheapest way to learn reinforcement learning:
- David Silver's RL lectures (free)
- implement multi-armed bandits
- code Q-learning from scratch
- implement DQN on CartPole
- learn policy gradients and PPO
- train an agent in Gymnasium
- read the GRPO paper and understand how it relates to LLM post-training
you can learn most RL fundamentals on a laptop.
resources in replies.
Fabryka AI retweeted
亚历山大·王（Alexandr Wang）在谈到“如何拉开与 99% 人的差距”时，给出了极其朴素甚至残酷的答案：
顶层竞争根本不拼智商，拼的是极端执行力的非对称倾斜。
他的三条核心准则：
1. 绝对聚焦，压榨认知带宽： 绝大多数人的忙碌只是“低密度伪勤奋”。真正的专注是把全部精力砸在唯一的核心支点上，拒绝任何无关干扰。
2. 把“过度交付”做成护城河： 行业标准要求走 1 英里，你就走完 10 英里。当所有人都按合同完成 100% 时，常态化交付 200% 的人会直接卷走全部声誉与溢价。
3. 带靶向的超额输出，拒绝无目的内耗： 努力不是盲目自我感动，而是针对具体目标的持续过载测试。把每一次交付都当成生死决战来打。
商业世界最公平的一点在于：
聪明人到处都是，但能忍受极度枯燥并长期超额交付的人凤毛麟角。
执行力溢出，就是最暴力的护城河。
Fabryka AI retweeted
Foundation models are trained in stages: pretraining, midtraining, and post-training. But is this separation fundamental?
We introduce unified training: a single algorithm for the full pipeline. This gives
⚙️ Simpler infrastructure
🔬 A framework for better science
(1/n)
Fabryka AI retweeted
Researchers built simulated worlds where physics is quietly broken. Heavier objects fall slower. Gravity oscillates. There's a speed limit.
Then they let 12 LLMs explore with no instructions on what to look for. Would they notice, and work out the new law through their own experiments?
11 of 12 models discovered nothing. The best, GPT-6-Astra, reached a 16.7% discovery rate, and even then it often pinned down the altered law only partially.
The framework is called LongChang. The gap it exposes is curiosity: models left anomalies uninvestigated.
arxiv.org/abs/2610.37649
#AI #LLM #AIResearch #AIForScience
Fabryka AI retweeted
I would love to believe this, and genuinely hope it's a breakthrough
However, I'm pretty bearish on it for a number of reasons
- people have tried training on bytes many times; examples include ByT5, CANINE, Perceiver, Byte Latent Transformer (BLT), and many more
- none have succeeded in replacing a tokenizer + Transformer, the main reason being that these are too complex to work with in practice.
- Hierarchical networks are no different: they consist of various (5 !!) submodules, which are just way too hard to serve at scale
- They use State-Space Machine (SSM) layers, none of which are adopted by major AI labs so far due to the complexity, although Qwen-4 might be different
- So far, the bitter lesson has taught us that simpler architectures typically win in the end
I've always believed that intelligence should come from simple, general building blocks that learn their own abstractions directly from raw data; in other words, the bitter lesson. I think we're finally seeing a real step towards AI that can learn from all data.
Language modeling on raw bytes is a microcosm of this ideal, and to me it's a central problem in AI: if we can't build LLMs that learn end-to-end from bytes, how will we handle the rest of the world's data that we understand even less?
Last year we introduced hierarchical networks (H-Nets) as a first step. Our new version finds up to 3-5x efficiency gains in controlled scaling law experiments, and at larger scale matches an open-source model trained on 11T tokens with ~10x less data and compute. The posts below walk through what we found and why we think it works.
Fabryka AI retweeted
I interviewed ~2 dozen researchers, engineers, research / engineer managers etc. asking them how they / their teams use coding agents in their work. Then I sprinkled it with my opinions and hot takes and wrote a substack post (link in the next tweet, I'd be delighted if you subscribe; of course, the post is free and any future post will be free too!) trying to triangulate the current reality of software engineering / industry research as opposed to what gets peddled on social media.
TL;DR (3. and 4. are the most important!):
1. Models, harnesses, workflows: don't bother with fancy workflows, they don't really work. Use stock Claude Code / Codex (or Copilot with Claude / GPT models if you are a Microsoft stan), perhaps with subagents in some very special cases. Write a design doc, then plan a change, then implement it, then merge. Wash, rinse, repeat. Don't get swayed by people online who are trying to sell you snake oil in the form of their new harness or workflow. None of this stuff works (in general).
2. Correctness/tests/reading or not reading code: it's OK not to read code if you do one of the following three things. a) One-off stuff / demos, which you need to work on exactly one input. b) Testable / verifiable stuff, e.g., GPU kernels. c) Stuff, where it's OK to be "directionally correct" (ML / data work / research code). In other cases, especially if you require any degree of maintainability / reliability or your code is customer-facing, reading code is still necessary. Reading tests and steering the model to write better tests is a good idea too. It's often good to use a different model to review the code that the other model wrote. Even if you approve the PRs without changing them, reading still helps for future more productive prompting.
3. Team dynamics: if we are not careful, agents can really undermine team dynamics. Two biggest problems are "slop infestation", where over time all PRs / emails / Slack messages end up being unreadable slop, and "overbuilding", where we are building and shipping useless stuff to get recognition or as a form of procrastination. Other, less severe but still important problems include: reduced communication, being secretive about ideas, decreased diversity of thinking, mode collapse of the ideas, trusting LLMs more than people.
4. Call for action: we really need to be careful with preserving team work and collaboration. We still need to "form packs to hunt mammoths", but with coding agents this becomes increasingly harder. Overall, many of the agents-induced problems stem in myopic short-term thinking of the sort "we have a couple of years left until our jobs are fully automated, so I can as well extract as much value as possible, everyone else be damned". While somewhat understandable, I think it’s important to resist these urges, if nothing else, for your own sanity’s sake.
[Full disclosure: until 2 months ago, I worked at OpenAI and my work directly contributed to Codex, I'm a big fan of coding agents and quite strongly AI-pilled, but there is little I hate more than fearmongering, gaslighting, lies and rage-bait that are ubiquitous in this space, unfortunately, hence spending time on all of this.]
Fabryka AI retweeted
Didn’t see anyone mention this so guess I will
My team and I looked into H-Nets pretty seriously when we first started exploring LLM efficiency
It’s genuinely super interesting, and what they’ve done is very impressive, but I’m still skeptical of the inference economics at scale
Dynamic chunking means you deal with unpredictable compute per request, which hence means harder batching and more complicated GPU scheduling/memory management
But there are ways to handle this by using separate queues and grouping requests by expected workloads + custom kernels, but i don’t think it’s sufficient to beat the competitions
A hybrid architecture might be interesting though!
Because you can reduce FLOPs on paper and still end up with worse throughput, higher latency and more expensive inference in prod
At the end of the day what really matters is how many requests you can serve per GPU at the same quality and latency
But again
That really doesn’t take anything from what @_albertgu and his team have achieved, we gave up on H-nets due to the above and a few other reasons but regardless, it still has very significant use cases even if it struggles to scale
Gg
I've always believed that intelligence should come from simple, general building blocks that learn their own abstractions directly from raw data; in other words, the bitter lesson. I think we're finally seeing a real step towards AI that can learn from all data.
Language modeling on raw bytes is a microcosm of this ideal, and to me it's a central problem in AI: if we can't build LLMs that learn end-to-end from bytes, how will we handle the rest of the world's data that we understand even less?
Last year we introduced hierarchical networks (H-Nets) as a first step. Our new version finds up to 3-5x efficiency gains in controlled scaling law experiments, and at larger scale matches an open-source model trained on 11T tokens with ~10x less data and compute. The posts below walk through what we found and why we think it works.
Fabryka AI retweeted
I'm posting a series of prediction questions from Stanford CS312 (Deep Learning Alchemy) exams to test deep learning intuition.
First up: optimization. If we increase an LM's batch size 8× (same token budget), can rescaling learning rate or weight decay compensate? Try below ↓
This quiz is part of our optimizer hyperparameter assignment and unit designed by @wen_kaiyue
Setup: We train an 8-layer, ~35M-parameter Transformer on 614.4M tokens using AdamW. At batch size 64, the best reference run has peak LR 0.003, weight decay (WD) 0.2, and final validation loss 2.9178.
Now increase the batch size from 64 to 512, which gives 8× fewer optimizer steps. Everything else stays fixed except the changes below.
Which recipe gives the lowest final validation loss?
A. Keep LR and WD unchanged
B. LR ×√8
C. LR ×8
D. WD ×8
E. LR ×√8 and WD ×√8
What would you predict, and why?
Are you a SWE / ML engineer and u want to learn robotics and get access to real hardware to connect to your agents and get real feedback?
Fabryka AI retweeted
Cartesia scaled up H-Nets, and one of the most beautiful emergent behaviors is in-context chunking!
Learned tokenization matters if we want to build intelligence for modalities that BPE handles poorly: DNA, proteins, time series, sensor data, molecules, audio, video, images.
I've always believed that intelligence should come from simple, general building blocks that learn their own abstractions directly from raw data; in other words, the bitter lesson. I think we're finally seeing a real step towards AI that can learn from all data.
Language modeling on raw bytes is a microcosm of this ideal, and to me it's a central problem in AI: if we can't build LLMs that learn end-to-end from bytes, how will we handle the rest of the world's data that we understand even less?
Last year we introduced hierarchical networks (H-Nets) as a first step. Our new version finds up to 3-5x efficiency gains in controlled scaling law experiments, and at larger scale matches an open-source model trained on 11T tokens with ~10x less data and compute. The posts below walk through what we found and why we think it works.
Fabryka AI retweeted
Completely absurd improvements. Herman is unbelievably good at optimizing.. I remember him crushing almost every CS336 leaderboard, and now he's broken NanoGPT.
Also somewhat surprised that big gains in nanoGPT have come from AI assistance of humans instead of autoresearch.
