pinto/Vault
1
1
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Initial commit

Browse Source
This commit is contained in:
twotalesanimation
2026-06-11 10:46:09 +02:00
commit 81ad7e4ea9
223 changed files with 39530 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
transcoder-remote/Dockerfile
+44
View File
@@ -0,0 +1,44 @@
# syntax=docker/dockerfile:1
# ---------------------------------------------------------------------------
# Stage 1 – Build TypeScript
# ---------------------------------------------------------------------------
FROM node:20-alpine AS builder
WORKDIR /app
COPY package*.json ./
RUN npm install
COPY tsconfig.json ./
COPY src/ ./src/
RUN npm run build
# ---------------------------------------------------------------------------
# Stage 2 – Runtime image
# ---------------------------------------------------------------------------
FROM node:20-alpine AS runner
# Install FFmpeg (includes ffprobe).
RUN apk add --no-cache ffmpeg
WORKDIR /app
# Production dependencies only.
COPY package*.json ./
RUN npm install --omit=dev
# Compiled output from builder stage.
COPY --from=builder /app/dist ./dist
# Non-root user for defence-in-depth.
RUN addgroup -g 1001 -S transcoder \
&& adduser -u 1001 -S transcoder -G transcoder
# Work directory the worker uses for temp files.
RUN mkdir -p /work && chown transcoder:transcoder /work
USER transcoder
CMD ["npm", "run", "start"]
transcoder-remote/README.md
+196
View File
@@ -0,0 +1,196 @@
# Remote Transcoder Worker
Runs on **Windows Docker Desktop** (or any machine with Docker).
Pulls video jobs from the Hetzner CMS over HTTPS, transcodes them locally using
all available CPU cores, and pushes the finished HLS package back — no SSH,
no shared drives, no VPS CPU load.
---
## How it works
```
Docker Desktop (local) Hetzner VPS (CMS)
────────────────────── ─────────────────
1. POST /api/transcoder/claim → Lock next uploaded video
← { videoId, downloadUrl }
2. GET /api/transcoder/download/:id → Stream source MP4
3. ffmpeg (local CPU)
Generate HLS segments + playlists
4. Create ZIP of HLS output
5. POST /api/transcoder/upload/:id → Extract ZIP, validate, rename
Update status → "transcoded"
6. Repeat until queue empty, exit 0
```
---
## Prerequisites
| Requirement | Notes |
|---|---|
| Docker Desktop | Windows, Mac, or Linux |
| CMS environment variable `TRANSCODER_SECRET` | Add to CMS `.env` and redeploy |
| CMS redeployed with new API routes | See **CMS changes** section |
---
## Setup
### 1 — Add `TRANSCODER_SECRET` to the CMS
In your Hetzner CMS `.env` (or however you manage secrets):
```env
TRANSCODER_SECRET=replace-with-a-strong-random-secret
```
Generate a secret with:
```bash
openssl rand -hex 32
```
Redeploy the CMS so the new API routes are live.
### 2 — Configure the worker
```bash
cd transcoder-remote
cp .env.example .env
# Edit .env and fill in CMS_URL and TRANSCODER_SECRET
```
`.env` example:
```env
CMS_URL=https://cms.yourdomain.com
TRANSCODER_SECRET=replace-with-a-strong-random-secret
```
---
## Running
### Build and run (one command)
```powershell
docker compose up --build
```
The container will:
1. Build the TypeScript worker
2. Start processing all queued videos
3. Exit automatically when the queue is empty
### Run again later (no rebuild)
```powershell
docker compose up
```
### Run without docker-compose
```powershell
docker build -t transcoder-remote .
docker run --rm --env-file .env transcoder-remote
```
---
## Log output
```
[Worker] Remote Transcoder starting
[Worker] CMS : https://cms.yourdomain.com
[Worker] Work dir: /work
[Job] cm1a2b3c4d5e6f7g8h9i0j
[Download] Starting...
[Download] 312 MB received...
[Download] 624 MB in 38s
[Probe] Resolution: 1920x1080
[HLS] Creating 1080p
[HLS] Creating 720p
[HLS] Creating 480p
[Transcode] 1080p, 720p, 480p in 8m 14s
[Package] Creating zip...
[Package] 1.1 GB in 22s
[Upload] Starting...
[Upload] Done in 31s
[Complete] cm1a2b3c4d5e6f7g8h9i0j in 9m 45s
[Worker] Queue empty. Jobs processed this run: 1
[Worker] Exiting.
```
---
## CMS changes (already applied)
Four new API routes were added to the CMS Next.js app:
| Route | Purpose |
|---|---|
| `POST /api/transcoder/claim` | Atomically claim next job |
| `GET /api/transcoder/download/:videoId` | Stream source MP4 |
| `POST /api/transcoder/upload/:videoId` | Receive HLS zip |
| `POST /api/transcoder/fail/:videoId` | Mark job failed |
All routes require `Authorization: Bearer <TRANSCODER_SECRET>`.
The claim endpoint uses `SELECT … FOR UPDATE SKIP LOCKED` so multiple workers can run concurrently without racing on the same job.
---
## Disk space requirements
Each job requires roughly:
| Step | Space |
|---|---|
| Downloaded MP4 | Up to ~2 GB |
| HLS output (all variants) | ~1–4× source size |
| ZIP archive | ~same as HLS output |
| **Total per job** | **~4–8 GB** |
Ensure Docker Desktop's virtual disk limit is large enough (Settings → Resources → Disk image size). 80 GB+ is recommended for large educational videos.
Temp files are deleted automatically after each job.
---
## Transcoding settings
Identical to the VPS transcoder — output is fully compatible with the existing
HLS player:
| Variant | Resolution | Video bitrate | Audio |
|---|---|---|---|
| 1080p | 1920×1080 | 3500k (max 4000k) | AAC 128k |
| 720p | 1280×720 | 1800k (max 2000k) | AAC 128k |
| 480p | 854×480 | 900k (max 1000k) | AAC 128k |
Variants are skipped if the source resolution is smaller than the preset.
---
## Environment variables
| Variable | Required | Description |
|---|---|---|
| `CMS_URL` | ✅ | Public HTTPS base URL of the CMS |
| `TRANSCODER_SECRET` | ✅ | Shared secret matching `TRANSCODER_SECRET` on CMS |
| `WORK_DIR` | optional | Temp directory inside container (default: `/work`) |
---
## Exit codes
| Code | Meaning |
|---|---|
| `0` | Success (all jobs processed, or no jobs found) |
| `1` | Fatal error (missing env vars, DB unreachable, etc.) |
transcoder-remote/docker-compose.yml
+18
View File
@@ -0,0 +1,18 @@
services:
transcoder:
build:
context: .
dockerfile: Dockerfile
# Load all variables from .env in this directory.
env_file: .env
environment:
WORK_DIR: /work
# Expose /work to the Docker host so you can inspect temp files during
# development. Remove this volume in production for fully ephemeral runs.
volumes:
- transcoder_work:/work
# The worker exits when the job queue is empty – do not restart it.
restart: "no"
volumes:
transcoder_work:
transcoder-remote/package.json
+20
View File
@@ -0,0 +1,20 @@
{
"name": "transcoder-remote",
"version": "1.0.0",
"description": "Remote HLS transcoder worker – runs on local desktop, transcodes videos from the Hetzner CMS via HTTPS API",
"main": "dist/index.js",
"scripts": {
"start": "node dist/index.js",
"build": "tsc",
"dev": "ts-node --esm src/index.ts"
},
"dependencies": {
"archiver": "^6.0.2"
},
"devDependencies": {
"@types/archiver": "^6.0.3",
"@types/node": "^20.0.0",
"ts-node": "^10.9.2",
"typescript": "^5.3.3"
}
}
transcoder-remote/src/api-client.ts
+133
View File
@@ -0,0 +1,133 @@
// src/api-client.ts
// Typed HTTP client for all CMS transcoder API endpoints.
import * as fssync from "fs";
import * as fs from "fs/promises";
import { createReadStream, createWriteStream } from "fs";
import { Readable } from "stream";
import { pipeline } from "stream/promises";
export interface JobInfo {
videoId: string;
downloadUrl: string;
}
export class TranscoderApiClient {
constructor(
private readonly cmsUrl: string,
private readonly secret: string
) {}
private get authHeader(): Record<string, string> {
return { Authorization: `Bearer ${this.secret}` };
}
/**
* Atomically claim the next available job.
* Returns null when the queue is empty.
*/
async claimJob(): Promise<JobInfo | null> {
const res = await fetch(`${this.cmsUrl}/api/transcoder/claim`, {
method: "POST",
headers: this.authHeader,
});
if (!res.ok) {
const body = await res.text();
throw new Error(`Claim failed: HTTP ${res.status} – ${body}`);
}
return res.json() as Promise<JobInfo | null>;
}
/**
* Stream the source MP4 from the CMS directly to a local file path.
* Calls onProgress with the total bytes received so far (optional).
* Returns the total bytes downloaded.
*/
async downloadVideo(
downloadUrl: string,
destPath: string,
onProgress?: (bytesReceived: number) => void
): Promise<number> {
const res = await fetch(downloadUrl, { headers: this.authHeader });
if (!res.ok) {
const body = await res.text();
throw new Error(`Download failed: HTTP ${res.status} – ${body}`);
}
if (!res.body) {
throw new Error("Download response had no body");
}
const writeStream = createWriteStream(destPath);
let totalBytes = 0;
const nodeReadable = Readable.fromWeb(
res.body as unknown as import("stream/web").ReadableStream<Uint8Array>
);
nodeReadable.on("data", (chunk: Buffer) => {
totalBytes += chunk.length;
onProgress?.(totalBytes);
});
await pipeline(nodeReadable, writeStream);
return totalBytes;
}
/**
* Stream a ZIP archive to the CMS upload endpoint.
* The CMS extracts the archive, validates it, and marks the video transcoded.
*/
async uploadHls(videoId: string, zipPath: string): Promise<void> {
const stat = await fs.stat(zipPath);
const readStream = createReadStream(zipPath);
// Node.js native fetch requires duplex: 'half' when body is a stream.
const res = await (fetch as typeof fetch)(
`${this.cmsUrl}/api/transcoder/upload/${videoId}`,
{
method: "POST",
headers: {
...this.authHeader,
"Content-Type": "application/zip",
"Content-Length": stat.size.toString(),
},
body: Readable.toWeb(
readStream
) as unknown as BodyInit,
// @ts-expect-error – required for streaming body in Node 18+ fetch
duplex: "half",
}
);
if (!res.ok) {
const body = await res.text();
throw new Error(`Upload failed: HTTP ${res.status} – ${body}`);
}
}
/**
* Tell the CMS that this job failed unrecoverably.
*/
async markFailed(videoId: string, errorMessage: string): Promise<void> {
try {
const res = await fetch(`${this.cmsUrl}/api/transcoder/fail/${videoId}`, {
method: "POST",
headers: { ...this.authHeader, "Content-Type": "application/json" },
body: JSON.stringify({ error: errorMessage }),
});
if (!res.ok) {
console.error(
`[markFailed] HTTP ${res.status} for ${videoId}: ${await res.text()}`
);
}
} catch (err) {
// Best-effort – don't throw if the fail call itself errors.
console.error(`[markFailed] Could not reach CMS for ${videoId}:`, err);
}
}
}
transcoder-remote/src/index.ts
+175
View File
@@ -0,0 +1,175 @@
// src/index.ts
// Remote transcoder worker entry point.
//
// Workflow for each job:
// 1. Claim job via CMS API
// 2. Download source MP4
// 3. Transcode to HLS (FFmpeg – identical to VPS transcoder)
// 4. Zip the HLS output
// 5. Upload zip to CMS
// 6. Repeat until queue is empty, then exit 0.
import * as fs from "fs/promises";
import * as path from "path";
import { TranscoderApiClient } from "./api-client";
import { transcodeToHls } from "./transcoder";
import { createZipArchive } from "./packager";
// ---------------------------------------------------------------------------
// Environment validation
// ---------------------------------------------------------------------------
function requireEnv(name: string): string {
const val = process.env[name];
if (!val) {
console.error(`[Fatal] Missing required environment variable: ${name}`);
process.exit(1);
}
return val;
}
const CMS_URL = requireEnv("CMS_URL").replace(/\/$/, "");
const TRANSCODER_SECRET = requireEnv("TRANSCODER_SECRET");
const WORK_DIR = (process.env.WORK_DIR ?? "/work").replace(/\/$/, "");
// ---------------------------------------------------------------------------
// Formatting helpers
// ---------------------------------------------------------------------------
function fmtBytes(bytes: number): string {
if (bytes >= 1024 ** 3) return `${(bytes / 1024 ** 3).toFixed(1)} GB`;
if (bytes >= 1024 ** 2) return `${(bytes / 1024 ** 2).toFixed(0)} MB`;
return `${(bytes / 1024).toFixed(0)} KB`;
}
function fmtDuration(ms: number): string {
const s = Math.floor(ms / 1000);
const m = Math.floor(s / 60);
const h = Math.floor(m / 60);
if (h > 0) return `${h}h ${m % 60}m ${s % 60}s`;
if (m > 0) return `${m}m ${s % 60}s`;
return `${s}s`;
}
async function safeRm(...paths: string[]): Promise<void> {
for (const p of paths) {
await fs.rm(p, { recursive: true, force: true }).catch(() => {});
}
}
// ---------------------------------------------------------------------------
// Single job handler
// ---------------------------------------------------------------------------
async function processJob(
client: TranscoderApiClient,
videoId: string,
downloadUrl: string
): Promise<void> {
const jobStart = Date.now();
const jobDir = path.join(WORK_DIR, videoId);
const inputMp4 = path.join(jobDir, "input.mp4");
const hlsDir = path.join(jobDir, "hls");
const zipPath = path.join(jobDir, "output.zip");
console.log(`\n[Job] ${videoId}`);
try {
await fs.mkdir(hlsDir, { recursive: true });
// 1. Download --------------------------------------------------------
console.log("[Download] Starting...");
const dlStart = Date.now();
let lastReported = 0;
const downloadedBytes = await client.downloadVideo(
downloadUrl,
inputMp4,
(received) => {
if (received - lastReported >= 100 * 1024 * 1024) {
lastReported = received;
process.stdout.write(`\r[Download] ${fmtBytes(received)} received...`);
}
}
);
process.stdout.write("\n");
console.log(
`[Download] ${fmtBytes(downloadedBytes)} in ${fmtDuration(Date.now() - dlStart)}`
);
// 2. Transcode -------------------------------------------------------
console.log("[Transcode] Starting...");
const txStart = Date.now();
const result = await transcodeToHls(inputMp4, hlsDir);
const txMs = Date.now() - txStart;
console.log(
`[Transcode] ${result.variants.join(", ")} in ${fmtDuration(txMs)}`
);
// 3. Package ---------------------------------------------------------
console.log("[Package] Creating zip...");
const pkgStart = Date.now();
const zipBytes = await createZipArchive(hlsDir, zipPath);
console.log(
`[Package] ${fmtBytes(zipBytes)} in ${fmtDuration(Date.now() - pkgStart)}`
);
// 4. Upload ----------------------------------------------------------
console.log("[Upload] Starting...");
const ulStart = Date.now();
await client.uploadHls(videoId, zipPath);
console.log(`[Upload] Done in ${fmtDuration(Date.now() - ulStart)}`);
console.log(`[Complete] ${videoId} in ${fmtDuration(Date.now() - jobStart)}`);
} finally {
// Always clean up local work directory.
await safeRm(jobDir);
}
}
// ---------------------------------------------------------------------------
// Main loop
// ---------------------------------------------------------------------------
async function main(): Promise<void> {
console.log("[Worker] Remote Transcoder starting");
console.log(`[Worker] CMS : ${CMS_URL}`);
console.log(`[Worker] Work dir: ${WORK_DIR}`);
await fs.mkdir(WORK_DIR, { recursive: true });
const client = new TranscoderApiClient(CMS_URL, TRANSCODER_SECRET);
let processed = 0;
while (true) {
const job = await client.claimJob();
if (!job) {
if (processed === 0) {
console.log("\n[Worker] No jobs queued – nothing to do.");
} else {
console.log(`\n[Worker] Queue empty. Jobs processed this run: ${processed}`);
}
break;
}
try {
await processJob(client, job.videoId, job.downloadUrl);
processed++;
} catch (err) {
const msg = err instanceof Error ? err.message : String(err);
console.error(`[Error] ${job.videoId}: ${msg}`);
await client.markFailed(job.videoId, msg);
}
}
console.log("[Worker] Exiting.");
}
main().catch((err) => {
console.error("[Fatal]", err);
process.exit(1);
});
transcoder-remote/src/packager.ts
+39
View File
@@ -0,0 +1,39 @@
// src/packager.ts
// Creates a flat ZIP archive of an HLS output directory.
// Video segments (.ts) are already compressed, so zlib level 0 (store only)
// is used to avoid wasting CPU on content that won't compress further.
import archiver from "archiver";
import { createWriteStream } from "fs";
/**
* Zip the contents of sourceDir (flat – no parent prefix inside the zip).
* @returns Total bytes written to the zip file.
*/
export function createZipArchive(
sourceDir: string,
outputZipPath: string
): Promise<number> {
return new Promise((resolve, reject) => {
const output = createWriteStream(outputZipPath);
const archive = archiver("zip", {
zlib: { level: 0 }, // store-only; .ts segments are already compressed
});
output.on("close", () => resolve(archive.pointer()));
archive.on("error", reject);
archive.on("warning", (err) => {
if (err.code === "ENOENT") {
console.warn("[Packager] Warning:", err.message);
} else {
reject(err);
}
});
archive.pipe(output);
// false = files land at the root of the zip, not inside a subdirectory.
archive.directory(sourceDir, false);
archive.finalize();
});
}
transcoder-remote/src/transcoder.ts
+165
View File
@@ -0,0 +1,165 @@
// src/transcoder.ts
// FFmpeg + HLS logic – identical behaviour to the VPS transcoder.
// Keeps all preset values, FFmpeg flags, and playlist structure unchanged
// so output is fully compatible with the existing HLS player.
import { spawn } from "child_process";
import * as fs from "fs/promises";
import * as path from "path";
// ---------------------------------------------------------------------------
// Presets – must stay in sync with the VPS transcoder.
// ---------------------------------------------------------------------------
const HLS_PRESETS = [
{ name: "1080p", width: 1920, height: 1080, bitrate: "3500k", maxrate: "4000k" },
{ name: "720p", width: 1280, height: 720, bitrate: "1800k", maxrate: "2000k" },
{ name: "480p", width: 854, height: 480, bitrate: "900k", maxrate: "1000k" },
] as const;
type HlsPreset = (typeof HLS_PRESETS)[number];
// ---------------------------------------------------------------------------
// Subprocess helpers
// ---------------------------------------------------------------------------
function runFFmpeg(args: string[]): Promise<void> {
return new Promise((resolve, reject) => {
const proc = spawn("ffmpeg", args, { stdio: "inherit" });
proc.on("error", reject);
proc.on("close", (code) => {
if (code === 0) resolve();
else reject(new Error(`FFmpeg exited with code ${code}`));
});
});
}
export async function probeResolution(
input: string
): Promise<{ width: number; height: number }> {
return new Promise((resolve, reject) => {
const proc = spawn("ffprobe", [
"-v", "error",
"-select_streams", "v:0",
"-show_entries", "stream=width,height",
"-of", "csv=s=x:p=0",
input,
]);
let output = "";
proc.stdout.on("data", (d: Buffer) => { output += d.toString(); });
proc.stderr.on("data", () => {/* suppress */});
proc.on("error", reject);
proc.on("close", (code) => {
if (code !== 0) return reject(new Error(`ffprobe exited with code ${code}`));
const parts = output.trim().split("x").map(Number);
if (parts.length !== 2 || !parts[0] || !parts[1]) {
return reject(new Error(`Could not parse resolution: "${output.trim()}"`));
}
resolve({ width: parts[0], height: parts[1] });
});
});
}
// ---------------------------------------------------------------------------
// Per-variant HLS stream
// ---------------------------------------------------------------------------
async function createVariant(
input: string,
outputDir: string,
preset: HlsPreset
): Promise<void> {
const segmentPattern = path.join(outputDir, `${preset.name}_%03d.ts`);
const playlistPath = path.join(outputDir, `${preset.name}.m3u8`);
const args = [
"-y",
"-i", input,
"-c:v", "libx264",
"-preset", "medium",
"-profile:v", "main",
"-crf", "20",
"-vf", `scale=w='min(${preset.width},iw)':h='min(${preset.height},ih)':force_original_aspect_ratio=decrease:force_divisible_by=2`,
"-b:v", preset.bitrate,
"-maxrate", preset.maxrate,
"-bufsize", "4000k",
"-c:a", "aac",
"-b:a", "128k",
"-f", "hls",
"-hls_time", "6",
"-hls_playlist_type", "vod",
"-hls_flags", "independent_segments",
"-hls_segment_filename", segmentPattern,
playlistPath,
];
console.log(`[HLS] Creating ${preset.name}`);
await runFFmpeg(args);
}
// ---------------------------------------------------------------------------
// Master playlist
// ---------------------------------------------------------------------------
async function createMasterPlaylist(
outputDir: string,
variants: readonly string[]
): Promise<void> {
const lines: string[] = ["#EXTM3U", "#EXT-X-VERSION:3"];
for (const name of variants) {
const preset = HLS_PRESETS.find((p) => p.name === name)!;
const bandwidth = parseInt(preset.maxrate.replace("k", ""), 10) * 1000;
lines.push(
`#EXT-X-STREAM-INF:BANDWIDTH=${bandwidth},RESOLUTION=${preset.width}x${preset.height}`
);
lines.push(`${preset.name}.m3u8`);
}
await fs.writeFile(path.join(outputDir, "master.m3u8"), lines.join("\n"));
}
// ---------------------------------------------------------------------------
// Public API
// ---------------------------------------------------------------------------
export interface TranscodeResult {
/** Names of variants that were generated, e.g. ["1080p","720p","480p"]. */
variants: string[];
}
/**
* Transcode inputPath to HLS inside outputDir.
* outputDir will be created if it does not exist.
* Behaviour is identical to the VPS transcoder.
*/
export async function transcodeToHls(
inputPath: string,
outputDir: string
): Promise<TranscodeResult> {
const { width, height } = await probeResolution(inputPath);
console.log(`[Probe] Resolution: ${width}x${height}`);
const allowed = HLS_PRESETS.filter(
(p) => p.width <= width && p.height <= height
);
if (allowed.length === 0) {
throw new Error(
`No suitable HLS variants for source resolution ${width}x${height}`
);
}
await fs.mkdir(outputDir, { recursive: true });
for (const preset of allowed) {
await createVariant(inputPath, outputDir, preset);
}
const variantNames = allowed.map((p) => p.name);
await createMasterPlaylist(outputDir, variantNames);
return { variants: variantNames };
}
transcoder-remote/tsconfig.json
+15
View File
@@ -0,0 +1,15 @@
{
"compilerOptions": {
"target": "ES2022",
"module": "commonjs",
"lib": ["ES2022"],
"outDir": "dist",
"rootDir": "src",
"strict": true,
"esModuleInterop": true,
"skipLibCheck": true,
"resolveJsonModule": true
},
"include": ["src"],
"exclude": ["node_modules", "dist"]
}