Overview
This is the story of offloading the heavy AI work in my Shorts Clipper tool (a self-hosted long-video-to-vertical-shorts generator) from CPU on my main Docker host onto a spare Windows box with a GPU. The two expensive steps—speech-to-text transcription and video cropping/encoding—were both running on CPU, which is slow, and I had a GPU sitting idle on another machine.
Background
The setup involves a small Python HTTP server (gpu-whisper-server) running directly on the Windows box. This server does two jobs: faster-whisper transcription using CUDA instead of CPU, and video crop/encode using ffmpeg with NVENC (hardware video encoding) instead of software encoding. The Linux clipper pipeline just calls it over plain HTTP with a long read timeout since some jobs take a while.
How It Works
SSH Access Setup
- Windows OpenSSH ignores the normal per-user authorized_keys file for any account in the local Administrators group; the key has to go into a special admin-only authorized_keys file instead.
- Password auth was disabled server-side for automation purposes, so it had to be public-key-only from the start.
- I tried creating a dedicated low-privilege account just for automation but gave up when the account didn't have a real profile until it logged in interactively. Used the existing admin account's already-working access instead.
- Processes started via SSH get killed once the SSH command returns, so anything that needs to keep running has to go through Windows Task Scheduler.
Server Fixes
- The Whisper model gets lazily downloaded from Hugging Face on first request, which could cause timeouts during restarts. Fixed by setting a permanent environment variable for faster downloads.
- Changed the source video upload strategy to only upload once per job and cut multiple clips concurrently.
- A subtle env-var bug left crop offload turned off, so only transcription was being offloaded. Once that got flipped on, an NVENC "could not open encoder" error occurred due to a genuine GPU driver issue. Fixed by updating the driver.
- The crop math can produce odd pixel dimensions, which H.264 rejects. Fixed by flooring to the nearest even number.
- A stray leftover process from an earlier crash was causing health checks to fail.
Deployment and Reliability
- Hit a "which copy is the source of truth" mistake mid-deploy—edited a stale mirrored copy of the server code instead of the actual git repo.
- Bigger picture reliability lesson: the GPU server is just a Windows Scheduled Task tied to an interactive login, not a real always-on service. Jobs failed for hours in a row because the process had silently died with nothing else showing signs of trouble. The fix was noticing and restarting it manually via the scheduled task.
Results
Once all of that was sorted, I verified the whole pipeline end-to-end on real footage: downloaded a video, transcribed on GPU, picked highlights, cropped multiple clips concurrently on GPU, and confirmed the output files were correct videos (right codec, right dimensions, right duration).
Lessons Learned
- Windows OpenSSH has specific requirements for key authentication.
- Processes started via SSH need to be managed by Task Scheduler if they need to keep running after the SSH connection closes.
- Nested quoting through multiple shells can break silently; it's more reliable to write a .ps1 script and run it directly with PowerShell flags.
- Always diff against the real source before trusting a "fixed" file, especially when dealing with networked mirrors.
This setup significantly improved the performance of my Shorts Clipper tool by offloading heavy tasks to the GPU box.