Creating Latch's Demo with Claude & Ffmpeg

There's this idea that had been in my head for a while. Since GenAI is increasingly good at generating code, how can we use that in dashboards, to have charts on demand without engineers explicitly building every shape or form for the data UI yet meeting standards set in the application?

I talk about my research, how developers at Google are creating A2UI and how I built Latch and the principles I used this over at this article. Latch is live at latch.isaacantwi.com. You can should check it out.

When I was done, I wanted a walkthrough video but I did not want to narrate it myself, and I did not want to spend a day in an editor. A thought occurred to me: there's AI now, and this is small enough that it shouldn't take ten hours. Let's see how much of it I can hand off.

This is the walkthrough video I created and this article talks about how I did it.

Tools I used

1. Loom, to record the screen with the mic off
2. Claude Code, for the script, the timing, and the ffmpeg work
3. Edge-tts, a free command line tool that uses the voices from Edge's read-aloud feature
4. FFmpeg, to cut frames out of the video and mix the audio back in
5. Bensound, for the royalty free background music

How it all came together

The script came first

Before recording anything I had Claude write the script from the notes I already had on the project. The script contained what to click, what should be on screen, and what the voice says, which I reviewed and approved. The intial script Claude Code created sounded "like AI" 😂 I find that very annoying. It kept using terms like "real numbers", "genuinely" and that's what made a review very necessary. There's enough AI slop on the internet.

Recording without a mic

I recorded it in Loom without my camera or mic on following the script and pausing for a couple of seconds after each action so the voice would have room. However the recording drifted from the script in a few places, which turned out to be fine with some adjustments after.

Reading the video without watching it

Claude can't watch a video, but it can look at pictures. So it pulled one frame a second with ffmpeg and tiled them into contact sheets, twelve frames per sheet:

ffmpeg -i latch-loom.mp4 -vf "fps=1,scale=480:-1,tile=4x3" sheets/sheet_%02d.jpg

From those it could tell that the last card left the board at 0:19, the first answer landed at 0:32, the partner view appeared at 1:36, and the chat opened at 1:57. Every line got a window between two of those moments. A line that ran longer than its window got cut.

The Synthetic Voice Narration

edge-tts is free and needs no key. I started with a female voice, then switched to Andrew, a male US voice, which sounded more like someone showing their own work.

edge-tts --voice en-US-AndrewMultilingualNeural --text "..." --write-media 03.mp3

Each clip is delayed to its start time and mixed into one track, then laid under the original video with the picture untouched:

ffmpeg -i latch-loom.mp4 -i 01.mp3 -i 02.mp3 \
  -filter_complex "[1:a]adelay=500|500[a1];[2:a]adelay=13500|13500[a2];[a1][a2]amix=inputs=2:normalize=0[out]" \
  -map 0:v -map "[out]" -c:v copy latch-narrated.mp4

Music from Bensound

A Bensound track under the whole thing at one steady level, sixteen decibels below the voice, with a short fade at each end. I tried ducking first, where the music dips under speech and comes back up in the gaps. It sounded busier than the video needed, so it went.

Music I used: Bensound.com/royalty-free-music
License code: QKCZVPHTCRFZON0O
Artist: Marcus P.

Takeaways

None of this needed a second screen recoding take. For simple projects and explorations, communicating my idea shouldn't take another 10 hours. That's counter productive. By off loading these things to GenAI, I'm free to engage my thoughts and explore, and that's why I'd most likely do this again. Took a couple of minutes.

Ultimately, I think people need to interact with people. It's becoming increasingly annoying when AI slop is generated and given to the next person. Of course they can tell AI was used, and they don't see you in the work. But this is an ongoing debate. When should AI be used and when should humans do the work?