Frontier LLMs can create videos from code but understanding and editing videos is the hard part.
Frontier LLMs can already produce video by writing code. They write the markup, styles, and animation timelines that render to frames. Any coding agent can now turn a one-line prompt into a decent video.
Increasingly, code-generated video is not only about the code. It is an arrangement of code and media assets. This includes the footage and references from users, as well as images, clips, and audio from generative models. A human editor can watch a clip once and know what it contains. An agent must understand each asset’s content and metadata in detail before arranging the assets with code, and that understanding matters during the creative process.
The first problem was taste. Videos from coding agents often had stiff motion and a generic visual style. We closed that gap over many iterations in the HyperFrames product, teaching the agent what good video looks like from the professional videos it references and remixes. Then we turned to the next problem: helping a coding agent understand video well enough to close the loop from the result back to the code, and to do so efficiently.
Measuring before optimizing
General coding agents like Claude Code or Codex are designed for all-purpose programming, so they lack the scaffolding that video work requires. They can often complete the task, but usually only after several rounds of user feedback and a substantial amount of tokens.
To understand where that effort goes, we instrumented our benchmark runs, reviewed thousands of agent sessions step by step, and classified each step by its purpose.
A few findings stood out:
- Effort was not evenly distributed. It concentrated around patterns that recur across tasks.
- These inefficient patterns span the entire production process, from understanding source material and composing the frame, to aligning audio and reviewing the final cut.
- In some cases, the agent had to build its own temporary tooling before it could make progress.
- In others, even that tooling was not enough: the agent still could not see or hear the result precisely, and it kept iterating under uncertainty.
A closer look at any one of these patterns reveals a similar shape. Some approaches come naturally to the model but are expensive. The agent iterates heavily to verify each change, and estimates alignment instead of measuring it. Other methods it is unsure how to use, so it tries, guesses, and backtracks when a guess is wrong. A change that appears small often takes several of these loops.
These patterns became our engineering focus. We worked through them over repeated rounds of evaluation, design, and iteration, and each round started from the same question: what does the agent need to know at this step, and why does it have to build that knowledge itself?
Understanding what goes in and what comes out
Many tasks depend on multimodal understanding from the outset. The agent must analyze a reference’s structure, understand pacing and style, combine generated footage with code-driven graphics, or time motion to existing music and voiceover. If that understanding is incomplete, the first draft misses the user’s intention even when the code is correct.
Every revision depends on understanding too. The agent has to know how elements move and how audio and picture align, spot where visual artifacts appear, and work out which component a piece of feedback refers to. For purely code-driven visuals, this could in principle be derived from the source by simulating the entire render. Once external media is involved, much of it exists only in the assets and the rendered output.
Evolving with the underlying model
Our goal was simple: A coding agent built for video.
Because our agent writes code, it can produce anything code can express. Our harness adds what code alone cannot: a reliable understanding of picture and sound, applied throughout creation and revision.
The harness adds only what video requires. It does not override the agent’s general abilities or lock it into a fixed workflow, and decisions that belong to the model stay with the model. Gains in reasoning, planning, and coding carry straight into video work, and the agent no longer rebuilds its scaffolding for every task. The result is a more efficient harness that improves alongside the models it runs on.
Building a harness that evolves
Most agent harnesses today are already highly customizable. But as an agent matures, a lot of patches, policies, feature flags, and vertical capabilities accumulate in the harness. They end up scattered across the codebase, and their logic becomes entangled.
Over time, it becomes harder to isolate a single feature or module and measure its effect on the agent’s behavior, because the code and logic behind it are spread across several surfaces. Video work depends heavily on these kernel behaviors and requires extensive end-to-end agent evaluation.
We believe a good harness must be evaluation-driven. Each new generation of models brings changes in behavior, so some parts of an existing harness need to change and others should be removed. Those decisions should not rest on intuition. They should be made rigorously, through extensive evaluation, and efficiently enough to keep pace with each model release.
So when we built our new harness, we moved each policy, feature, and capability into its own standalone plugin. We also applied the same plugin approach to our existing harnesses, adding adapters and shared interfaces so that a single capability can run across all of them.
The benefit goes beyond flexibility. It lets us run frequent experiments on our internal benchmark: remove one or more plugins, then evaluate both the agent’s traces and the final videos end to end. Each result shows whether that component still earns its place in the harness.
It also changes how we begin new work. When we build a new harness, we do not start from scratch. We carry over the existing plugins that can be shared, along with what we have already learned about each one.
Evolving the harness
A plugin-based harness also changes how we improve the harness itself. Rather than maintaining a single harness and patching it over time, we maintain several harness checkpoints, similar to how models have checkpoints. Each checkpoint is a different combination of plugins, and each runs continuously against our internal benchmark.
Furthermore, an analysis system reviews every run, covering both the agent’s behavior at each step and the final video, recording the best-performing checkpoints. Self-improvement that optimizes for a single dimension tends to stall. Instead, we keep the best checkpoints on each dimension, so the next round can choose its direction from a broader view.
Initially, improvements were made by hand. Our engineers reviewed the analysis, identified where the agent was spending its effort, and tuned the plugins accordingly.
We then introduced a proposer agent. Starting from the best-performing checkpoints on each dimension, and guided by a set of principles and policies we maintain, it proposes plugins to add, modify, or remove. Because it draws on several dimensions at once, it avoids optimizing in a single direction and stalling there. Each proposal becomes a new checkpoint and is evaluated the same way as any other.
As the system matured, some features were proposed, evaluated, and shipped entirely by agents.
We see self-improvement, whether through search or learning, as a promising direction for agent harnesses. Each change is a self-contained unit that can be proposed, tested, and then kept or discarded independently.
Same model, same tasks, different harness
We compared a recent checkpoint of our harness with Claude Code. Both ran the same frontier model at the same effort with the same requests and assets. We ran them both across 17 video tasks spanning reference recreation, remixing, brand videos, library footage, and edits.
On 78% of tasks, our harness completed the same work with less effort. Across the full set, costs fell by about 30% and our HyperFrames agent needed about 35% fewer model turns. This gap was wider on tasks built around HyperFrames’ own workflows, such as remixing a reference video, where costs fell by 40% and the number of model turns fell by more than half.
These savings came with no reduction in quality. In blind pairwise evaluations, a judge model rated the two harnesses equally.
Looking ahead
What general coding agents can create today is just the starting point. Our goal is to raise the ceiling of what’s possible with code-generated video. That means helping the agent understand pictures and sound well enough to take on more ambitious projects and carrying a creator’s intent through each revision.
We’re now releasing this harness as our own custom agent in HyperFrames Studio. Next, we’ll extend it to more video workflows, including 3D, agentic editing, and finding footage that fits each individual project. Each new capability will go through the same cycle of evaluation and improvement.
