Initial commit
This commit is contained in:
twotalesanimation
@@ -0,0 +1,196 @@
|
||||
# Remote Transcoder Worker
|
||||
|
||||
Runs on **Windows Docker Desktop** (or any machine with Docker).
|
||||
Pulls video jobs from the Hetzner CMS over HTTPS, transcodes them locally using
|
||||
all available CPU cores, and pushes the finished HLS package back — no SSH,
|
||||
no shared drives, no VPS CPU load.
|
||||
|
||||
---
|
||||
|
||||
## How it works
|
||||
|
||||
```
|
||||
Docker Desktop (local) Hetzner VPS (CMS)
|
||||
────────────────────── ─────────────────
|
||||
1. POST /api/transcoder/claim → Lock next uploaded video
|
||||
← { videoId, downloadUrl }
|
||||
2. GET /api/transcoder/download/:id → Stream source MP4
|
||||
3. ffmpeg (local CPU)
|
||||
Generate HLS segments + playlists
|
||||
4. Create ZIP of HLS output
|
||||
5. POST /api/transcoder/upload/:id → Extract ZIP, validate, rename
|
||||
Update status → "transcoded"
|
||||
6. Repeat until queue empty, exit 0
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Prerequisites
|
||||
|
||||
| Requirement | Notes |
|
||||
|---|---|
|
||||
| Docker Desktop | Windows, Mac, or Linux |
|
||||
| CMS environment variable `TRANSCODER_SECRET` | Add to CMS `.env` and redeploy |
|
||||
| CMS redeployed with new API routes | See **CMS changes** section |
|
||||
|
||||
---
|
||||
|
||||
## Setup
|
||||
|
||||
### 1 — Add `TRANSCODER_SECRET` to the CMS
|
||||
|
||||
In your Hetzner CMS `.env` (or however you manage secrets):
|
||||
|
||||
```env
|
||||
TRANSCODER_SECRET=replace-with-a-strong-random-secret
|
||||
```
|
||||
|
||||
Generate a secret with:
|
||||
```bash
|
||||
openssl rand -hex 32
|
||||
```
|
||||
|
||||
Redeploy the CMS so the new API routes are live.
|
||||
|
||||
### 2 — Configure the worker
|
||||
|
||||
```bash
|
||||
cd transcoder-remote
|
||||
cp .env.example .env
|
||||
# Edit .env and fill in CMS_URL and TRANSCODER_SECRET
|
||||
```
|
||||
|
||||
`.env` example:
|
||||
```env
|
||||
CMS_URL=https://cms.yourdomain.com
|
||||
TRANSCODER_SECRET=replace-with-a-strong-random-secret
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Running
|
||||
|
||||
### Build and run (one command)
|
||||
|
||||
```powershell
|
||||
docker compose up --build
|
||||
```
|
||||
|
||||
The container will:
|
||||
1. Build the TypeScript worker
|
||||
2. Start processing all queued videos
|
||||
3. Exit automatically when the queue is empty
|
||||
|
||||
### Run again later (no rebuild)
|
||||
|
||||
```powershell
|
||||
docker compose up
|
||||
```
|
||||
|
||||
### Run without docker-compose
|
||||
|
||||
```powershell
|
||||
docker build -t transcoder-remote .
|
||||
docker run --rm --env-file .env transcoder-remote
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Log output
|
||||
|
||||
```
|
||||
[Worker] Remote Transcoder starting
|
||||
[Worker] CMS : https://cms.yourdomain.com
|
||||
[Worker] Work dir: /work
|
||||
|
||||
[Job] cm1a2b3c4d5e6f7g8h9i0j
|
||||
|
||||
[Download] Starting...
|
||||
[Download] 312 MB received...
|
||||
[Download] 624 MB in 38s
|
||||
|
||||
[Probe] Resolution: 1920x1080
|
||||
[HLS] Creating 1080p
|
||||
[HLS] Creating 720p
|
||||
[HLS] Creating 480p
|
||||
[Transcode] 1080p, 720p, 480p in 8m 14s
|
||||
|
||||
[Package] Creating zip...
|
||||
[Package] 1.1 GB in 22s
|
||||
|
||||
[Upload] Starting...
|
||||
[Upload] Done in 31s
|
||||
|
||||
[Complete] cm1a2b3c4d5e6f7g8h9i0j in 9m 45s
|
||||
|
||||
[Worker] Queue empty. Jobs processed this run: 1
|
||||
[Worker] Exiting.
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## CMS changes (already applied)
|
||||
|
||||
Four new API routes were added to the CMS Next.js app:
|
||||
|
||||
| Route | Purpose |
|
||||
|---|---|
|
||||
| `POST /api/transcoder/claim` | Atomically claim next job |
|
||||
| `GET /api/transcoder/download/:videoId` | Stream source MP4 |
|
||||
| `POST /api/transcoder/upload/:videoId` | Receive HLS zip |
|
||||
| `POST /api/transcoder/fail/:videoId` | Mark job failed |
|
||||
|
||||
All routes require `Authorization: Bearer <TRANSCODER_SECRET>`.
|
||||
|
||||
The claim endpoint uses `SELECT … FOR UPDATE SKIP LOCKED` so multiple workers can run concurrently without racing on the same job.
|
||||
|
||||
---
|
||||
|
||||
## Disk space requirements
|
||||
|
||||
Each job requires roughly:
|
||||
|
||||
| Step | Space |
|
||||
|---|---|
|
||||
| Downloaded MP4 | Up to ~2 GB |
|
||||
| HLS output (all variants) | ~1–4× source size |
|
||||
| ZIP archive | ~same as HLS output |
|
||||
| **Total per job** | **~4–8 GB** |
|
||||
|
||||
Ensure Docker Desktop's virtual disk limit is large enough (Settings → Resources → Disk image size). 80 GB+ is recommended for large educational videos.
|
||||
|
||||
Temp files are deleted automatically after each job.
|
||||
|
||||
---
|
||||
|
||||
## Transcoding settings
|
||||
|
||||
Identical to the VPS transcoder — output is fully compatible with the existing
|
||||
HLS player:
|
||||
|
||||
| Variant | Resolution | Video bitrate | Audio |
|
||||
|---|---|---|---|
|
||||
| 1080p | 1920×1080 | 3500k (max 4000k) | AAC 128k |
|
||||
| 720p | 1280×720 | 1800k (max 2000k) | AAC 128k |
|
||||
| 480p | 854×480 | 900k (max 1000k) | AAC 128k |
|
||||
|
||||
Variants are skipped if the source resolution is smaller than the preset.
|
||||
|
||||
---
|
||||
|
||||
## Environment variables
|
||||
|
||||
| Variable | Required | Description |
|
||||
|---|---|---|
|
||||
| `CMS_URL` | ✅ | Public HTTPS base URL of the CMS |
|
||||
| `TRANSCODER_SECRET` | ✅ | Shared secret matching `TRANSCODER_SECRET` on CMS |
|
||||
| `WORK_DIR` | optional | Temp directory inside container (default: `/work`) |
|
||||
|
||||
---
|
||||
|
||||
## Exit codes
|
||||
|
||||
| Code | Meaning |
|
||||
|---|---|
|
||||
| `0` | Success (all jobs processed, or no jobs found) |
|
||||
| `1` | Fatal error (missing env vars, DB unreachable, etc.) |
|
||||
Reference in New Issue
Block a user