yash

Video editing will change forever when people start sharing their edits on GitHub. I want a future where someone asks "how did you make this?" and you just send them a link to the project source that agents can also read and understand. The brief, project source, timeline, assets, effects, techniques, and rules. All of it. Then agents can use all of it to make edits for you. These projects could also become training examples, just like source code is used to train coding models. But there is a problem. AI is already good at many of these things individually. Understanding a brief, using tools, following editing techniques and applying them. They're also really good at creating them. But getting all of them on the same page is still hard. That is exactly why they are an excellent collaboration partner for a professional in the field, whereas inexperienced ones often produce rather sloppy results. The project context, production brief, editing methods, and your taste. The agent needs to follow all of these while making changes. Currently, you'll see that the agents are busy with tool calls, revisiting their steps, and focusing to execute the tool actions. They're missing and mixing the context at some point, and forget the real purpose of that given task. Instead, they should focus on the brief and result itself. They should spend more time on this part. That's where orchestration is missing. We already have plugins, extensions, presets, and tutorials for traditional video editors. People have spent years building these things. However, when you ask for a video edit, just because they are available doesn't mean an agent will find and use them anytime. Ask an agent to edit a video. You'll often need to tell it which tool to use, give it some methods, explain the steps, then check if it applied everything correctly. Furthermore, this will likely require hours of prompting and back and forth correspondence. Yes, agents can use Premiere, DaVinci, and After Effects. But at what cost? How much back and forth, just to apply something correctly with your help? We should focus on the purpose first. A clip or sequence needs cuts, effects, animations, and timing changes. So we need a source of truth that the agent can read easily, and a simple way to apply those changes. If the project source is code, the agent can work directly with that structure. It can see what's there and change it without having to do every step through the app's interface that take minutes for each. Look at how agents build a landing page or a dashboard. There are so many libraries, components, public repos, and examples they can use. They can check what they made, read the code, and change it. We need more of this for video editing, with the methods included. An effect or an SFX should come with examples and rules for when to use it. A scene should have a clear structure. The agent should have steps to follow and a way to check the result. We can put these into skills so we don't have to explain the whole process in every prompt. However, this is not a feasible method if you lack the necessary project context and cannot manage every detail with great precision. This is especially true given the absence of verification techniques, common in the world of code but absent in the world of video, and the fact that agents operate under the constraints of a very limited context, resources, and tool calls. This is why we decided to make the project's source of truth as code in Diffusion Studio. Agents can read your edits, change them, and repeat them. You can even turn your own techniques into skills for your video production workflow. For example, I just created a transition effect, then asked Claude Code to review it and apply it in the right places, following my structure and rules. Done. I gave it my technique in a form it could read and use. Now imagine more people sharing their edits with the source code. For example, imagine Sam Kolder (well known for his transition and editing techniques) sharing his own technique using this method. We could build open source versions of many editing methods, and also tools and effects we use through plugins today. People could share scenes, templates, components, and instructions. Others could improve them, and agents could access the code and use the relevant skills. These projects could also become useful examples for teaching agents. You have the brief, the code, the timeline, and the final output video together. The source shows the exact assets, prompts used for generative media, timing, settings, and steps behind the edit. They can improve it, repeat it, copy parts of it, change it, or try their own method. That also gives us clear examples for training and testing agents. Then, I think the quality can get much better. The agent has working methods, references, techniques, and examples to follow, and we can keep improving them with human feedback in an environment where proven methods, workflows and contribution space are used. We already have the right techniques and workflows in a different field. All we need to do is combine them for a different purpose. That's why we used this approach in our product. I think we'll see more editors, motion designers, and content creators sharing their own taste, editing recipes, and rules this way. Once an agent has access to those examples and skills, even a simple prompt could do much more.