Building an AI video pipeline with HyperFrames (and a lot of Cursor)
I built a small system that turns a talking-head avatar video into a short-form vertical edit: face on the bottom, visuals on top, captions locked to speech, B-roll when it actually matches the words, and motion graphics when the script earns them. This post is about how that thing is put together, and what it felt like to build it mostly inside Cursor with a mix of models and a lot of iteration.
No big manifesto. Just the architecture and the process.
What it actually does
You drop an avatar clip into a project folder, point a config at a backdrop (usually a YouTube URL), optionally throw some B-roll files into a folder with sensible names, and run:
npm run pipeline -- --project my-video
Out the other side you get a HyperFrames project, which includes HTML compositions, GSAP timelines, brand CSS, and captions that you can preview in the studio and render to MP4.
The layout is the classic short-form split: 1080x1920, face in the lower half, upper panel rotating through original footage, B-roll, and motion graphics. Captions sit on top. Ambient background and a little seam treatment keep the two halves from feeling glued together by accident.
That is the product. Underneath it is a staged TypeScript pipeline plus HyperFrames as the render engine.
The shape of the repo
I kept the workspace boring on purpose.
Root: shared tooling like the HyperFrames CLI, package.json, motion philosophy docs, and agent skills. pipeline/: the AI production pipeline (TypeScript, run with tsx). video-projects//: one folder per video. Avatar, B-roll, project.json, generated index.html, compositions, and renders.
That last part mattered. Early on it was tempting to dump everything at the root, but it gets messy fast. Each video is portable now: config, cache, and outputs are all in one place. You cd into the project and run HyperFrames from there. The pipeline never pretends it is Remotion. It is HTML-native video, GSAP timelines, and FFmpeg under the hood.
The six stages (this is the architecture)
Everything funnels through six stages. Each one writes artifacts into processed/ so you can re-run one piece without burning the whole run.
- Avatar prep: normalize the face video into a canonical WebM (processed/01-transparent.webm). Downstream stages always look there, so there is no guessing paths.
- Backdrop download: pull the original video with yt-dlp from project.json.
- Transcription: word-level transcript so captions and beat timing can latch onto real speech, not vibes.
- Plan visual beats: decide what shows up and when. Gemini helps plan motion-graphic moments, while B-roll is placed when a filename phrase actually appears in the transcript.
- Fulfill assets: turn that plan into real files, including the B-roll cache, motion-graphic HTML, and fulfilled beat JSON.
- Compose: stitch the HyperFrames project, including root index.html, captions, scaffold, and brand tokens.
The orchestrator is just pipeline/src/index.ts. Paths are centralized so stages do not invent their own folder layout. Caching is aggressive: if the artifact exists and the avatar has not changed, a stage can no-op. Swap the avatar and derived stuff invalidates automatically. Flip motion.regenerate when you want a fresh beat plan without rebuilding the face video again.
That cache story sounds dull until you have waited through a full re-encode for the fifth time. Then it feels like a feature.
The idea behind "beats"
I ended up thinking about the upper panel as three layers fighting for the same screen time:
Backdrop: the original video. Opens the piece and fills the gaps. B-roll: stuff you curated. The filename is the contract: if the stem matches a phrase in the transcript, it can land. If not, it gets skipped. No magical stock search required. Motion graphics: only when the speech has something visualizable, like a number, a short list, or a real comparison. Not every rhetorical "that sounds too easy."
A few rules that saved my sanity:
B-roll always wins. Don't trim a curated clip to make room for a graphic. MG share is a cap, not a quota. Zero motion graphics is a valid plan. Code backfills the obvious stats and lists if the model misses them. Prompts alone were too flaky. Upper-panel rotation is enforced in code so you don't get B-roll back-to-back with no breathing room.
There is also a creative MG path if you want Gemini to design the metaphor instead of filling one of the fixed recipes. It still lives inside a scaffold that owns the HyperFrames render contract, with templates as the fallback when the model output fails validation. It is fancy when it works and boring when it doesn't, which is intentional.
Libraries and tools I leaned on
Nothing exotic:
HyperFrames: compositions, studio preview, lint, render. The whole delivery surface. GSAP: timelines inside those HTML comps. One paused timeline per composition, registered the way HyperFrames expects. TypeScript + tsx: pipeline code you can run without a heavy build step. FFmpeg / fluent-ffmpeg / ffprobe: encode, probe durations, cache B-roll to the right panel size. Google Gemini: beat planning (Flash for planning, a heavier model when the creative MG pass is on). ElevenLabs: word-level timestamps via a local transcribe helper. yt-dlp: backdrop download. Playwright: handy for eyeballing frames when you are iterating on layout.
And then the soft tooling: Cursor for almost all of the writing and refactoring, agent skills for HyperFrames authoring patterns, and a memory/ folder in the repo so the architecture doesn't live only in chat history. That last one sounds nerdy, but it paid for itself the first time I came back a week later and didn't have to re-explain the beat model to myself.
I bounced between models while building: Cursor's agents, Gemini in the pipeline, and Fable in the mix when I was exploring ideas. Honestly, the specific model names matter less than the loop: try something, watch the preview, fix the contract, write down the decision.
Building it in Cursor (the process)
I didn't design the whole pipeline on a whiteboard and then type it in. It grew the other way.
First, I got a single short-form project rendering by hand in HyperFrames with a face, captions, and something in the upper panel. Then I asked: what is painful enough to automate? Transcription and compose landed early. Beat planning came later, and it went through a few wrong personalities: too dense, too filler-y, too trusting of the LLM. Each time the fix was the same pattern: pull another rule out of the prompt and into TypeScript.
Cursor was good for that kind of work. I would keep the decision log open, paste a bad preview ("why is there a slam on a throwaway line?"), and carve a validator or a candidate scanner instead of yelling at the prompt again. Skills helped when I was touching compositions. HyperFrames has a specific render contract (data-start, class="clip", timeline registration, no animating video width directly). Missing those rules looks fine in your head and broken in the studio.
The rhythm that stuck:
- Change one stage or one rule.
- Run the pipeline on a real project (I have been using health explainer avatars as the test bed).
- Preview in HyperFrames before baking a render.
- If the upper panel feels wrong, fix the plan or the rotation, not just the CSS.
- Write a short note in memory/ so next-week-me doesn't undo it.
It is less "AI builds my videos" and more "AI proposes beats, code keeps them honest, HyperFrames ships the pixels."
What I'm happy about
A few things feel right:
Deterministic where it counts. Same avatar fingerprint, same transcript anchors, same filename matches. The model gets a lane, but it doesn't own the timeline. Projects stay self-contained. Hand someone a video-projects/ folder and they have the inputs and config story. Failure modes degrade. Creative MG falls back to templates. Missing B-roll matches are skips, not crashes. Sparse MG is allowed. The human still picks the B-roll. That was a deliberate choice. I would rather name a file after the phrase it belongs to than wire another stock API.
What's next (maybe)
I want the creative motion pass to feel more like a junior designer who actually read the line, and less like kinetic type restating the caption. The scaffold and cookbook path is a start. More recipe polish, better brand presets, tighter preview loops: all fair game.
If you are building something similar, start with one honest vertical edit you like, automate the boring stages first, and only then let a model touch timing. Timing is where "smart" systems get weird.
Anyway, that is the project. HyperFrames for the video surface, a six-stage TypeScript pipeline for the brains, and Cursor for the long middle where most of the real work happens.
If you build on top of it or tear a piece out for your own stack, I would love to hear what broke first. That is usually where the interesting architecture lives.
Thanks for hanging out.