Overview
In the spirit of continuous improvement and automation, I embarked on a project to build a self-hosted tool that could take any long video (think podcast, talk, or YouTube link) and turn it into several short vertical clips. The goal was to identify "viral" moments, crop them to 9:16 with face-tracking, add SEO-friendly titles/hooks, and optionally include captions or AI narration. This tool would run entirely locally without relying on any cloud services.
Background
I started this project on July 10, 2026, in my homelab directory ~/breadtoasting/shorts-clipper/. The initial codebase was based on the open-source project SamurAIGPT/AI-Youtube-Shorts-Generator, which I heavily customized to fit my needs. The tool is accessible via a password-gated web GUI at breadtoasting.com/clipper/, built using FastAPI and served by uvicorn.
How It Works
Input Handling
The Shorts Clipper can accept video inputs in three ways:
- URL Paste: Copy the URL of the long video directly into the tool.
- File Uploads: Use an input folder to drop files that need processing.
- Direct Uploads: For larger files, users can upload them directly through the web interface. However, uploads are capped at around 100MB due to Cloudflare's proxy limitations.
Core Pipeline
The core pipeline consists of several steps:
- Download or Accept an Upload: The tool either downloads a video from a provided URL or accepts a file upload.
- Transcription with Faster-Whisper: The video is transcribed using the faster-whisper library, which converts audio to text.
- LLM Highlight Moments and Write Titles/ Hooks: A local Large Language Model (Qwen2.5:7b via Ollama) processes the transcript to pick highlight moments and generate titles/hooks. This LLM uses an OpenAI-compatible API for seamless integration.
- FFmpeg + OpenCV Crop to 9:16 with Face Detection: The video is cropped to a 9:16 aspect ratio using FFmpeg, with face detection enabled to ensure the most engaging parts are captured.
- Serve/Download Finished Clips: Once processed, the clips can be downloaded or served directly from the tool.
The entire pipeline runs as a single background worker due to its CPU/GPU-intensive nature and long processing times per job.
Features Added Since Launch
Since the initial deployment, several features have been added:
- Burned-in Word-Level Captions: These are generated using faster-whisper's word-level timestamps and rendered via FFmpeg. They highlight the current word in bold with a white background.
- AI Narrator Voiceover: Users can opt for an AI-generated narrator voice that samples frames, describes scenes, writes scripts, and synthesizes them into audio using Piper, a local text-to-speech engine.
- Freemium/Multi-User Tier: The tool supports multiple invited users/accounts. Non-premium accounts get the breadtoasting.com outro appended to their clips, while premium accounts can opt out or in per job.
- Branded Outro Clip: An optional clip that gets appended during encoding.
- YouTube Auto-Publishing: Clips can be automatically published to multiple connected YouTube channel accounts with customizable titles/descriptions/tags.
Results
The tool has been a success, automating the process of creating engaging short clips from long videos. The combination of AI transcription, LLM-generated highlights, and custom captions or voiceovers ensures that each clip is both informative and visually appealing.
Lessons Learned
- Transcription Offload: One significant improvement was moving the transcription step off the main server to a dedicated GPU box on the LAN for faster processing.
- User Management: Implementing multi-user support with different tiers has been an interesting challenge, especially managing permissions and user accounts.
- Feature Prioritization: Deciding which features to prioritize (like YouTube auto-publishing vs. Instagram Reels) based on practicality and user demand.
Conclusion
Building the Shorts Clipper was a rewarding project that leveraged AI and automation to streamline video processing. The tool has proven useful for creating engaging content from long videos, and it continues to evolve with new features and optimizations.
Happy homelabbing!