twotalesanimation
197 lines
4.6 KiB
Markdown
197 lines
4.6 KiB
Markdown
# Remote Transcoder Worker
|
||
|
||
Runs on **Windows Docker Desktop** (or any machine with Docker).
|
||
Pulls video jobs from the Hetzner CMS over HTTPS, transcodes them locally using
|
||
all available CPU cores, and pushes the finished HLS package back — no SSH,
|
||
no shared drives, no VPS CPU load.
|
||
|
||
---
|
||
|
||
## How it works
|
||
|
||
```
|
||
Docker Desktop (local) Hetzner VPS (CMS)
|
||
────────────────────── ─────────────────
|
||
1. POST /api/transcoder/claim → Lock next uploaded video
|
||
← { videoId, downloadUrl }
|
||
2. GET /api/transcoder/download/:id → Stream source MP4
|
||
3. ffmpeg (local CPU)
|
||
Generate HLS segments + playlists
|
||
4. Create ZIP of HLS output
|
||
5. POST /api/transcoder/upload/:id → Extract ZIP, validate, rename
|
||
Update status → "transcoded"
|
||
6. Repeat until queue empty, exit 0
|
||
```
|
||
|
||
---
|
||
|
||
## Prerequisites
|
||
|
||
| Requirement | Notes |
|
||
|---|---|
|
||
| Docker Desktop | Windows, Mac, or Linux |
|
||
| CMS environment variable `TRANSCODER_SECRET` | Add to CMS `.env` and redeploy |
|
||
| CMS redeployed with new API routes | See **CMS changes** section |
|
||
|
||
---
|
||
|
||
## Setup
|
||
|
||
### 1 — Add `TRANSCODER_SECRET` to the CMS
|
||
|
||
In your Hetzner CMS `.env` (or however you manage secrets):
|
||
|
||
```env
|
||
TRANSCODER_SECRET=replace-with-a-strong-random-secret
|
||
```
|
||
|
||
Generate a secret with:
|
||
```bash
|
||
openssl rand -hex 32
|
||
```
|
||
|
||
Redeploy the CMS so the new API routes are live.
|
||
|
||
### 2 — Configure the worker
|
||
|
||
```bash
|
||
cd transcoder-remote
|
||
cp .env.example .env
|
||
# Edit .env and fill in CMS_URL and TRANSCODER_SECRET
|
||
```
|
||
|
||
`.env` example:
|
||
```env
|
||
CMS_URL=https://cms.yourdomain.com
|
||
TRANSCODER_SECRET=replace-with-a-strong-random-secret
|
||
```
|
||
|
||
---
|
||
|
||
## Running
|
||
|
||
### Build and run (one command)
|
||
|
||
```powershell
|
||
docker compose up --build
|
||
```
|
||
|
||
The container will:
|
||
1. Build the TypeScript worker
|
||
2. Start processing all queued videos
|
||
3. Exit automatically when the queue is empty
|
||
|
||
### Run again later (no rebuild)
|
||
|
||
```powershell
|
||
docker compose up
|
||
```
|
||
|
||
### Run without docker-compose
|
||
|
||
```powershell
|
||
docker build -t transcoder-remote .
|
||
docker run --rm --env-file .env transcoder-remote
|
||
```
|
||
|
||
---
|
||
|
||
## Log output
|
||
|
||
```
|
||
[Worker] Remote Transcoder starting
|
||
[Worker] CMS : https://cms.yourdomain.com
|
||
[Worker] Work dir: /work
|
||
|
||
[Job] cm1a2b3c4d5e6f7g8h9i0j
|
||
|
||
[Download] Starting...
|
||
[Download] 312 MB received...
|
||
[Download] 624 MB in 38s
|
||
|
||
[Probe] Resolution: 1920x1080
|
||
[HLS] Creating 1080p
|
||
[HLS] Creating 720p
|
||
[HLS] Creating 480p
|
||
[Transcode] 1080p, 720p, 480p in 8m 14s
|
||
|
||
[Package] Creating zip...
|
||
[Package] 1.1 GB in 22s
|
||
|
||
[Upload] Starting...
|
||
[Upload] Done in 31s
|
||
|
||
[Complete] cm1a2b3c4d5e6f7g8h9i0j in 9m 45s
|
||
|
||
[Worker] Queue empty. Jobs processed this run: 1
|
||
[Worker] Exiting.
|
||
```
|
||
|
||
---
|
||
|
||
## CMS changes (already applied)
|
||
|
||
Four new API routes were added to the CMS Next.js app:
|
||
|
||
| Route | Purpose |
|
||
|---|---|
|
||
| `POST /api/transcoder/claim` | Atomically claim next job |
|
||
| `GET /api/transcoder/download/:videoId` | Stream source MP4 |
|
||
| `POST /api/transcoder/upload/:videoId` | Receive HLS zip |
|
||
| `POST /api/transcoder/fail/:videoId` | Mark job failed |
|
||
|
||
All routes require `Authorization: Bearer <TRANSCODER_SECRET>`.
|
||
|
||
The claim endpoint uses `SELECT … FOR UPDATE SKIP LOCKED` so multiple workers can run concurrently without racing on the same job.
|
||
|
||
---
|
||
|
||
## Disk space requirements
|
||
|
||
Each job requires roughly:
|
||
|
||
| Step | Space |
|
||
|---|---|
|
||
| Downloaded MP4 | Up to ~2 GB |
|
||
| HLS output (all variants) | ~1–4× source size |
|
||
| ZIP archive | ~same as HLS output |
|
||
| **Total per job** | **~4–8 GB** |
|
||
|
||
Ensure Docker Desktop's virtual disk limit is large enough (Settings → Resources → Disk image size). 80 GB+ is recommended for large educational videos.
|
||
|
||
Temp files are deleted automatically after each job.
|
||
|
||
---
|
||
|
||
## Transcoding settings
|
||
|
||
Identical to the VPS transcoder — output is fully compatible with the existing
|
||
HLS player:
|
||
|
||
| Variant | Resolution | Video bitrate | Audio |
|
||
|---|---|---|---|
|
||
| 1080p | 1920×1080 | 3500k (max 4000k) | AAC 128k |
|
||
| 720p | 1280×720 | 1800k (max 2000k) | AAC 128k |
|
||
| 480p | 854×480 | 900k (max 1000k) | AAC 128k |
|
||
|
||
Variants are skipped if the source resolution is smaller than the preset.
|
||
|
||
---
|
||
|
||
## Environment variables
|
||
|
||
| Variable | Required | Description |
|
||
|---|---|---|
|
||
| `CMS_URL` | ✅ | Public HTTPS base URL of the CMS |
|
||
| `TRANSCODER_SECRET` | ✅ | Shared secret matching `TRANSCODER_SECRET` on CMS |
|
||
| `WORK_DIR` | optional | Temp directory inside container (default: `/work`) |
|
||
|
||
---
|
||
|
||
## Exit codes
|
||
|
||
| Code | Meaning |
|
||
|---|---|
|
||
| `0` | Success (all jobs processed, or no jobs found) |
|
||
| `1` | Fatal error (missing env vars, DB unreachable, etc.) |
|