Have fun!

Amherst, MA
Joined February 2012
My favorite artist has always been M.C. Escher. His "Print Gallery" has been hanging in my home for years. Recently I bumped into a 3B1B video about it that was really awesome and I thought it would be cool to generate such art. A bit out of my comfort zone, but luckily @Sophie0tq took the lead (🧵).
7
24
7
176
16,888
Introducing PC-ALM, a local-learning alternative to backpropagation. Our method trains 1000-layer neural nets using only local dynamics, and without backprop. Blog: pub.sakana.ai/pc-alm/ Standard deep learning relies on backpropagation. The brain, however, cannot implement backpropagation, at least not exactly. How can a physical system, such as the brain, solve multilayer credit assignment without explicit use of backprop? We look for inspiration in two related fields: distributed optimization and NeuroAI. In NeuroAI, predictive coding asks each neuron activation to solve an energy-based inference problem instead of using a standard forward pass. That inference step can be implemented as energy-minimization dynamics on local prediction errors. This perspective -- each layer as a dynamical system -- has proven promising, but performance of predictive coding hasn't scaled well with depth. Credit signals at far ends of the network struggle to diffuse into internal layers. We turn to distributed optimization, generalizing predictive coding to use an augmented Lagrangian instead of energy. This motivation stems back to a classic 1988 paper by LeCun, showing that the Lagrange multipliers of a deep network can be identified with gradients of a supervised loss. The augmented Lagrangian then bridges LeCun's perspective to the standard predictive coding that is used in NeuroAI. We find that this new perspective yields a natural PC-like alternative to backpropagation, resulting in a method we call PC-ALM. PC-ALM differs from PC in that it introduces dual neurons (Lagrange multipliers) as part of the layer-local dynamics, resulting in each layer acting as a PI feedback control system to minimize local prediction errors. We find that PC-ALM is capable of propagating signals to seemingly arbitrary depth, especially in deep narrow networks where standard PC struggles to learn. Ultimately, our motivation here is to understand how distributed physical systems, such as the brain, can compute credit signals using only local coupling and local dynamics. PC-ALM may also inform deep learning in neuromorphic hardware, where dynamics are cheaper than on GPUs. Paper: arxiv.org/abs/2605.31022 Code: github.com/SakanaAI/pc-alm
68
397
102
2,923
454,908
China’s humanoid robots are going to school — learning to fold, cook, and clean, guided by trainers in data-training centers.
172
527
116
2,264
324,667
Diffusion (or flow) makes for excellent policies, but training them with RL is notoriously hard: BPTT is unstable, RL over diffusion blows up the horizon. In our new paper, we show how we can optimize flow matching actors by using "one weird trick" -- "approximate" the Jacobian of the flow denoising process with the identity matrix. 👇
8
119
7
1,063
89,106
Just watched this AI-generated short film and yeah… anyone still saying AI can’t create something watchable is seriously behind. This isn’t “AI makes trash” anymore, this is real storytelling, and real potential. The people actually using these tools already know: AI isn’t replacing creativity, it’s leveling it up. Studios aren’t ignoring it – they’re evolving with it. Watch this and tell me AI hasn’t come a long way.
🤖 Made with AI
262
230
52
2,179
147,168
🎉 After one year of teamwork, we are excited to release our 3D foundation model — LingBot-Map! Unlike DA3/VGGT, LingBot-Map is a purely autoregressive model for streaming 3D reconstruction ⚡ It achieves ~20 FPS on 518×378 resolution over sequences exceeding 10,000 frames — and beyond 🚀 Two key insights behind LingBot-Map: 🔑 Keep SLAM's structural wisdom: build Geometric Context Attention with long-context modeling while maintaining a compact streaming state 🔑 Make everything end-to-end learnable — no optimization, no post-processing Let's check out our demos 👇
97
486
76
4,658
1,381,453
We built a real-time multiplayer game generated entirely by a neural network. MultiGen is a real-time multiplayer diffusion game engine that supports an arbitrary number of players (not just 2!) through a shared memory-based world model. Project website: ryanpo.com/multigen 1/5 🧵
8
33
6
226
22,816
What if your video generator could refine itself—at inference time? ❌No new models. ❌No retraining. ❌No external verifier. 💡 Introducing Self-Refining Video Sampling By reinterpreting a pretrained generator (Wan2.2, Cosmos) as a denoising autoencoder, we enable iterative self-refinement at inference time ➡️dramatically improving physical realism and achieving over 70% human preference! 🧵
3
25
8
180
42,710
I wrote an interactive article explaining the geometric intuition behind Rectified Flows. I visually explain why flow-models tend to learn curved trajectories, why this is bad for sampling latency, and a relatively simple technique for mitigating it. Check it out! Link 👇
18
161
19
1,242
159,553
Farewell, Rotary Position Embedding (RoPE). We foresee that RoPE will no longer be utilized in future LLMs. Something significant has already been revealed; the answer is within this image. Stay tuned! 🚀
11
5
4
202
73,349
What if next-token prediction wasn't a single forward pass, but a tiny optimization problem? Introducing: nanoEBM a tiny transformer that learns to think harder by doing gradient descent on its own predictions. You can start training on your Mac now - it comes < 400 lines
21
73
11
715
69,729
It is now possible to create 360 equirectangular VR 8K seamless videos from a single image! 😃 @TheoMediaAI @theworldlabs #aivideo #VR
32
57
11
611
52,435
From live video to 3D avatar in seconds. This #SIGGRAPH2025 paper from Adobe Research reconstructs a realistic head avatar on the fly, with no pre-cached data, and adapts seamlessly to facial motion for VR, animation, and online communication. 🔗 adobe.ly/4lyowVF
2
26
4
164
29,277
Yang retweeted
StreamDiT Real-Time Streaming Text-to-Video Generation StreamDiT enables real-time text-to-video generation at 16 FPS on a single GPU (H100)
7
52
9
321
43,705
Why does your RL training always collapse? In our new paper of RAGEN, we explore what breaks when you train LLM *Agents* with multi-turn reinforcement learning—and possibly how to fix it. 📄 github.com/RAGEN-AI/RAGEN/bl… 🌐 ragen-ai.github.io/ 1/🧵👇
8
89
16
431
101,160
🚨 Introducing VEGGIE 🥦—a unified, end-to-end, and versatile instructional video generative model. Current video editing methods struggle with: 1. Understanding direct user instructions 2. Handling diverse editing skills in one model 3. balancing multiple training objectives. VEGGIE solves these via: 1. Multimodal LLM for instruction understanding & reasoning 2. Video Diffusion Model for instruction-aligned edits. 3. End-to-end optimization using only diffusion loss. VEGGIE supports 8 skills, from object addition/removal/changing, background change & stylization to concept grounding & reasoning segmentation, outperforms SoTA methods, and exhibits zero-shot multimodal instructional & in-context video editing. arxiv.org/abs/2503.14350 🧵
2
40
3
92
24,780
Yang retweeted
Temporal Regularization Makes Your Video Generator Stronger
4
22
2
148
20,647
this is a very counterintuitive result
13
81
5
866
90,712
I've been excited about this for a while: a simple architectural change to the residual connection that allows arbitrary overlapping of computation of one layer and the communication of another layer, leading to ~30% speedup in TP! More on MoE and expert parallel to come soon!
Running your model on multiple GPUs but often found the speed not satisfiable? We introduce Ladder-residual, a parallelism-aware architecture modification that makes 70B Llama with tensor parallelism ~30% faster! Work done at @togethercompute. Co-1st author with @MayankMish98 and mentored by @ben_athi, @tri_dao arxiv.org/pdf/2501.06589 🧵[1/7]
3
62
1
500
52,227