This post is about the pipeline behind it: what worked, what didn't, and what each step actually costs.

The naive version doesn't work

My first idea was the obvious one: send the video to a multimodal model and ask "describe every shot".

The problem is sampling. Most video-capable models look at roughly one frame per second. A typical product video on TikTok cuts every 0.5 to 1.5 seconds, so the model misses shots, merges them, and invents timestamps. For a "breakdown" product, wrong timestamps are a dealbreaker.

So the rule became: measure with classic tools first, then let the model describe what was measured.

The breakdown pipeline

download → shot detection → speech-to-text → OCR → beat/BPM → keyframes → LLM → JSON template

1. Shot detection (PySceneDetect). Gives frame-accurate cut points, which also give a pacing metric: cuts per 10 seconds.
2. Speech-to-text with word-level timestamps, so the voiceover lines up with the shots.
3. OCR (tesseract) on keyframes. A lot of hooks are written on screen, not spoken, so skipping this loses the most important part of the video.
4. BPM by autocorrelation. I originally let the model guess the BPM. It was confidently wrong almost every time. Now the BPM is measured, and if it can't be measured it's simply left out.
5. Keyframes, capped at 12. The first 3 seconds and shots with on-screen text get priority. Going from "all shots" to 12 keyframes cut the cost of a 35-shot video from $0.052 to $0.030 with no visible drop in description quality.
6. The LLM (Gemini Flash) gets the cut list, transcript, OCR text and keyframes, and returns JSON that has to match a fixed schema: hook type, beat structure, one entry per measured shot, pacing, style and the variables a new product would fill in.

Because shots come from the detector, the model can't merge or invent them. It only labels them.

What it costs (measured)

A 19-second TikTok pasted by a user finished in 26.8 seconds for $0.0198. That's cheap enough that breakdowns on the site are free and need no sign-up. Videos over 3 minutes are rejected at the probe step, before any paid call happens.

Remixing: templates, not generated code

The second half is turning a template into a new video for someone else's product.

It was tempting to let an LLM write Remotion code at runtime. I didn't, for two reasons: it's slow and it fails in ways that are hard to retry safely. Instead:

• A set of offline Remotion templates (product demo listicle, before/after split, UGC hook captions, Reddit story split, green screen commentary...) is written and tested by hand.
• At runtime the LLM only produces parameters: the script, the text for each shot, timings, and which template to use.
• Remotion renders. A deterministic check afterwards looks for wrong dimensions, black or white frames, and text in the platform's unsafe zones.

Measured: one script plus 3 rendered videos with different hooks cost about $0.038 and took about 2 minutes. Each render is around $0.013; almost all the cost is the script.

Things that broke

• TTS controlled the video length. The voiceover set the duration, so a 19-second, 17-shot video came back with a third of the cuts. Fix: split each beat into fast sub-cuts. Pacing went from about 3 back to 6.3–6.8 cuts per 10 seconds.
• Everything got a voiceover. ASMR clips and music-only edits came out as AI narration. Now if the original has no speech, the remix has none either.
• "Why did it go viral" is not answerable from one video. Structure can be extracted reliably (hooks and script around 90%, shots around 80–85% in my checks). The reason a specific video blew up depends on the account, timing and the algorithm. That needs statistics across many videos, not one model call, so I stopped pretending otherwise in the UI.
• Copying content vs. copying structure. The script prompt is told to keep the structure and write new content, and two automatic checks reject made-up numbers and claims like "#1" or "guaranteed". If a check fails, it retries.

Try it

If you want to see what the JSON template looks like for a real video, paste a TikTok link at fypzo.com. Breakdowns are free and don't need an account.

I'd love feedback from anyone who has worked on video understanding: is there a better way to get shot boundaries on heavily edited clips (whip pans, zoom cuts) than a content detector?