Jiteng Mu (@JitengMu) profile picture

@Adobe Firefly. Ph.D. @UCSD ; M.S. @JohnsHopkins ; Intern @Nvidia @Adobe

Joined March 2018
Photo by Jiteng Mu (@JitengMu)Photo by Jiteng Mu (@JitengMu)Photo by Jiteng Mu (@JitengMu)Photo by Jiteng Mu (@JitengMu)Photo by Jiteng Mu (@JitengMu)Photo by Jiteng Mu (@JitengMu)Photo by Jiteng Mu (@JitengMu)Photo by Jiteng Mu (@JitengMu)Photo by Jiteng Mu (@JitengMu)Photo by Jiteng Mu (@JitengMu)
When an image-editing agent fails, does it really need another try? Introducing our new paper: EditAgent: Rethinking Agentic Image Editing. We argue that editing agents are bounded by two spaces: 🔧 Action space: what the agent can do. 🔍 Observation space: what its judge can see. Three findings: 1️⃣ Expanding the action space matters. Across 600 expert-annotated candidate sets, FLUX.2-klein-9B produced no acceptable candidate in 42.7%. Specialized methods rescued 175 of those 256 failures. 2️⃣ Expand what the judge can observe. EditAgent enriches the observation space with parallel candidates for comparison, task-specific verification evidence, and earlier images and requests as historical context—supporting better candidate selection and detecting when a new edit undoes an earlier one. 3️⃣ Explore diverse alternatives through parallel search. Different specialists, generalists, and repeated samples offer different ways to fulfill a request. Parallel search lets the judge compare these alternatives, while generating each candidate from the same source avoids carrying errors from one attempt into the next. Even with a single editor and the same four-generation budget, experts preferred parallel search over sequential refinement in 78.5% of non-tied comparisons—with at most half as many judge calls. EditAgent expands both spaces and connects them through parallel candidate generation and comparative judging. It improves two editing backbones on three benchmarks, and its accepted sessions provide training data that strengthens standalone editors. Our takeaway: building reliable editing agents requires expanding both what they can do and what their judges can observe.
8
6
1
23
33,624
I will join Tsinghua University, College of AI, as an Assistant Professor in the coming month. I am actively looking for 2026 spring interns and future PhDs (ping me if you are in #NeurIPS). It has been an incredible journey of 10 years since I attended an activity organized by Tsinghua University and decided to change my undergraduate major from Economics to Computer Science, inspired by one of the teammates. During the 10 years, I met with appreciation of many wonderful researchers/professors who led me to continued growth. 🐿️ My research focus will continue to be AI & Robotics, with a specific emphasis on Interactive Embodied Intelligence. You can check my homepage to learn more: maureenzou.github.io/lab.htm…. I am currently local to San Diego and will be attending #NeurIPS. Please ping me over WeChat or Email if any old or new friends are interested in having a coffee chat! (Really looking forward to meeting as many friends as possible at #NeurIPS) [The photo is one of the places that I will miss a lot in the US]
69
81
13
1,091
113,497
How to generate billion-scale manipulation demonstrations easily? Let us leverage generative models! 🤖✨ We introduce Dex1B, a framework that generates 1 BILLION diverse dexterous hand demonstrations for both grasping 🖐️and articulation 💻 tasks using a simple C-VAE model.
15
90
16
374
73,506
Test-Time Training (TTT) is now on Video! And not just a 5-second video. We can generate a full 1-min video! TTT module is an RNN module that provides an explicit and efficient memory mechanism. It models the hidden state of an RNN with a machine learning model, which is updated via gradient descent. Combined with a Diffusion Transformer, we are able to generate a 1-min Tom and Jerry cartoon. Enjoy our video with input script (not seen before): Jerry happily eats cheese in a tidy kitchen until Tom playfully takes it away, teasing him. Annoyed, Jerry packs his belongings and leaves home, dragging a small suitcase behind him. Later, Tom notices Jerry's absence, feels sad, and follows Jerry's tiny footprints all the way to San Francisco. Jerry sits disheartened in an alleyway, where Tom finds him, gently offering cheese as an apology. Jerry forgives Tom, accepts the cheese, and the two return home together, their friendship restored.
29
162
40
1,205
186,786
Excited to come out of stealth at @reveimage! Today's text-to-image/video models, in contrast to LLMs, lack logic. Images seem plausible initially but fall apart under scrutiny: painting techniques don't match, props don't carry meaning, and compositions lack intention. (1/4)
The Halfmoon 🌓 reveal: Congratulations to @reveimage on creating the world’s leading image generation model with Reve Image! Reve Image has been in the Artificial Analysis Image Arena over the past week and is the clear leader, beating strong competition including Recraft V3, Google's Imagen 3, Midjourney v6.1, and Black Forest Lab's FLUX.1.1 [pro]. The model stands out for its impressive text rendering, prompt adherence, and aesthetics. Currently, the model is accessible through Reve’s website (link below), an API is not yet available. See below for example images and a link to try the model out for yourself 🧵
41
83
20
823
150,926
Jiteng Mu retweeted
Meet our first general-purpose robot at @DexmateAI dexmate.ai/vega Adjustable height from 0.66m to 2.2m: compact enough for an SUV, tall enough to reach those impossible high shelves. Powerful dual arms (15lbs payload each) and omni-directional mobility for ultimate versatility. More videos showcasing Vega in action coming this week! Stay tuned to see this robot partner.
Introducing Vega: 🤖 @DexmateAI's newest robot that makes complex manipulation tasks simple. ✨ A step closer to intelligence and automation. 🚀 🎥 Watch now: youtu.be/PecqfiJNwQI #Robotics #AI #Automation
14
36
6
221
33,256
🐅 Want to rig your favorite meme character? Try “RigAnything: Template-Free Autoregressive Rigging for Diverse 3D Assets”! ✨RigAnything is a transformer-based model that sequentially generates skeletons without predefined templates. It creates high-quality skeletons for shapes in any pose, completing rigging in under 2 seconds per shape. 🧵(1/n)
20
220
28
1,253
129,839
Introducing “Diffusion Autoencoders are Scalable Image Tokenizers” (DiTo). We show that with proper designs and scaling up, diffusion autoencoders (a single L2 loss) can outperform the GAN-LPIPS tokenizers (hybrid losses) used in current SOTA generative models. (1/4)
6
100
7
495
90,923
Autoregressive models are picking up for image generation, but how about image editing? Given an image and a language description, we train one Autoregressive Transformer to do ANY EDITING.
Do we need separate models for image editing and conditional generation? We introduce🌟EditAR, a unified autoregressive model for diverse tasks, e.g., image editing, depth-to-image, edge-to-image, segmentation-to-image. jitengmu.github.io/EditAR/ More details🧵👇
7
12
80
13,734
Human videos are scaling up vision-language navigation! 🚀 It has been a rewarding journey to tame the VLM into the VLA model and enable it to learn from real-world human touring videos🚶🏽. Watching robots move through all kinds of places, following our instructions, and doing it all WITHOUT A MAP is amazing! Checkout our page: navila-bot.github.io
Without any maps and prior knowledge of the scene, our humanoid and quadruped can now navigate with human language instructions to anywhere outdoors and in any house we go!🔥🔥🔥 Introducing NaVILA, a 2-level navigation foundation model (mid-level action VLA + locomotion skills) leveraging massive offline datasets, e.g., not only sim envs, but also human touring videos. A thread 1/N
17
1
59
14,235
We're excited to introduce our new 1-step image generator, Diffusion2GAN at #ECCV2024, which enables ODE-preserving 1k image generation in just 0.16 seconds! Check out our #ECCV2024 paper mingukkang.github.io/Diffusi… and stop by poster #181 (Wed Oct 2, 10:30-12:30 CEST) if you're around:) This work was done in collaboration with @rzhang88, Connelly Barnes, Sylvain Paris, Suha Kwak, Jaesik Park, @elishechtman, @junyanz89, and Taesung Park Notably, we have one outstanding AC (Suha) and two outstanding reviewers (@minguk_kang and Taesung)! 1/
6
48
4
276
20,978
Jiteng Mu retweeted
🙌🤳
136
2,304
330
19,052
423,293
Cannot believe this finally happened! Over the last 1.5 years, we have been developing a new LLM architecture, with linear complexity and expressive hidden states, for long-context modeling. The following plots show our model trained from Books scale better (from 125M to 1.3B) than Mamba and Transformer, and our 1.3B model works better and better with longer context. The idea, Test-Time Training (TTT), is something we have been working on for over 5 years. I still remember when I started as a postdoc, discussing with Alyosha what to work on, he asked me to talk to Yu Sun on TTT. And that is when everything begins. Although this TTT is quite a different thing now. It is a network layer, replacing the hidden state of an RNN with a machine learning model. Instead of using a feature vector to represent a memory, our TTT layer maintains a small neural network to compress the input tokens. As each token comes in sequentially, the compression is done via gradient descent on this token to update the small neural network. This is currently applied to language modeling, but imagine applying it to videos. In the future, when modeling long videos, instead sampling 1 FPS, we can sample the frames densely, and these dense frames would be a burden for Transformer but a blessing for TTT layer. As they are essentially just augmentations in time to train a better network inside TTT. Please check: arxiv.org/abs/2407.04620
20
266
29
1,568
262,372
Introduce Open-𝐓𝐞𝐥𝐞𝐕𝐢𝐬𝐢𝐨𝐧🤖: 
⁣ We need an intuitive and remote teleoperation interface to collect more robot data. 𝐓𝐞𝐥𝐞𝐕𝐢𝐬𝐢𝐨𝐧 lets you immersively operate a robot even if you are 3000 miles away, like in the movie 𝘈𝘷𝘢𝘵𝘢𝘳. Open-sourced!
46
224
81
1,135
329,808
GPT4-V can describe the location via text, but can’t accurately output the coordinate of each word. Introducing: Pixel Aligned Language Models. It generates image captions along with the aligned pixel coordinates of the image. arxiv.org/abs/2312.09237 (1/n
7
56
6
297
65,351
Image Neural Field Diffusion Models is presented today at #CVPR2024 by @YinboChen . It is a simple idea that combines the Latent Diffusion Model with Implicit Functions, allowing high-resolution / any-resolution image generation. Below is a text-to-image result with 2k resolution. page: yinboc.github.io/infd/ paper: arxiv.org/abs/2406.07480
3
15
3
92
21,182
🤖SpatialRGPT can perceive and reason about "far/close", "high/low". Empowered by spatial understanding ability, it could figure out the number of floors.
🌟Introducing "🤖SpatialRGPT: Grounded Spatial Reasoning in Vision Language Model" anjiecheng.me/SpatialRGPT SpatialRGPT is a powerful region-level VLM that can understand both 2D and 3D spatial arrangements. It can process any region proposal (e.g., boxes or masks) and provide answers to complex spatial reasoning questions. 🧵(1/n)
2
8
24
5,635
🌟Introducing "🤖SpatialRGPT: Grounded Spatial Reasoning in Vision Language Model" anjiecheng.me/SpatialRGPT SpatialRGPT is a powerful region-level VLM that can understand both 2D and 3D spatial arrangements. It can process any region proposal (e.g., boxes or masks) and provide answers to complex spatial reasoning questions. 🧵(1/n)
10
110
10
464
102,607
Tesla Optimus can arrange batteries in their factories, ours can do skincare (on @QinYuzhe)! We opensource Bunny-VisionPro, a teleoperation system for bimanual hand manipulation. The users can control the robot hands in real time using VisionPro, flexible like a bunny. 🐇 github.com/Dingry/BunnyVisio… We also have kitchen tasks, playing Rubik's Cube, and dynamic motion tasks. Imitation learning policies are trained on sweeping with a broom, serving a drink, and wiping glasses. Check our website for more details: dingry.github.io/projects/bu… The project is led by Runyu Ding @dngxngxng3, Yuzhe Qin @QinYuzhe , and Jiyue Zhu
11
62
18
319
91,098
Cool work from @JitengMu on image editing! This looks like the future of Photoshop! Just select a few patches and move it, and boom, you get the edited picture! This enables the new control ability of the diffusion model: Only change the part you want to change. jitengmu.github.io/Editable_…
We introduce🌟Editable Image Elements🥳, a new disentangled and controllable latent space for diffusion models, that allows for various image editing operations (e.g., move, resize,  de-occlusion, object removal, variations, composition) jitengmu.github.io/Editable_… More details🧵👇
11
42
9,676