pinto/Vault
1
1
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Initial commit

Browse Source
This commit is contained in:
twotalesanimation
2026-06-11 10:46:09 +02:00
commit 81ad7e4ea9
223 changed files with 39530 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
transcoder/README.md
+339
View File
@@ -0,0 +1,339 @@
# HLS Transcoder Service
A scalable HLS (HTTP Live Streaming) transcoding service for converting MP4 videos into adaptive bitrate HLS streams using FFmpeg.
## Features
- ✅ Automatic video detection and transcoding
- ✅ Multiple quality variants (1080p, 720p, 480p)
- ✅ Master playlist generation for adaptive bitrate streaming
- ✅ Concurrent job processing
- ✅ Database integration with Prisma
- ✅ Comprehensive error handling and logging
- ✅ Docker support with multi-stage builds
- ✅ Health checks and graceful shutdown
## Prerequisites
### For Local Development
- Node.js 18+
- FFmpeg
- PostgreSQL
- TypeScript
### For Docker
- Docker Desktop (with Windows Support)
- Docker Compose 2.0+
## Quick Start
### Development Mode
```bash
# Install dependencies
cd transcoder
npm install
# Set environment variables
export DATABASE_URL="postgresql://cms_user:changeme@localhost:5432/cms_db"
export UPLOADS_DIR="/path/to/uploads"
export NODE_ENV="development"
# Run in development mode
npm run dev
# Or build and run
npm run build
npm start
```
### Docker Compose
```bash
# From project root
docker-compose up -d
# View logs
docker-compose logs -f transcoder
# Stop services
docker-compose down
```
## Configuration
### Environment Variables
| Variable | Description | Default |
|----------|-------------|---------|
| `DATABASE_URL` | PostgreSQL connection string | (required) |
| `UPLOADS_DIR` | Path to uploads directory | `/uploads` |
| `POLL_INTERVAL` | Check for new videos every N ms | `5000` |
| `CONCURRENT_JOBS` | Videos to transcode simultaneously | `2` |
| `NODE_ENV` | Environment (development/production) | `production` |
### Example Configuration
```bash
DATABASE_URL=postgresql://cms_user:secure_password@db.example.com:5432/cms_db
UPLOADS_DIR=/mnt/storage/uploads
POLL_INTERVAL=10000
CONCURRENT_JOBS=4
```
## Output Format
### Directory Structure
```
/uploads/hls/
├── {video_id_1}/
│ ├── master.m3u8 # Master playlist
│ ├── 1080p.m3u8 # 1080p variant
│ ├── 1080p_000.ts # TS segments
│ ├── 1080p_001.ts
│ ├── 720p.m3u8 # 720p variant
│ ├── 720p_000.ts
│ └── 480p.m3u8 # 480p variant
└── {video_id_2}/
└── ...
```
### Master Playlist Format
```m3u8
#EXTM3U
#EXT-X-VERSION:3
#EXT-X-STREAM-INF:BANDWIDTH=3500000,RESOLUTION=1920x1080
1080p.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=1800000,RESOLUTION=1280x720
720p.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=900000,RESOLUTION=854x480
480p.m3u8
```
## API Reference
### Database Schema
```prisma
model Video {
id String @id
transcodingStatus String @default("uploaded")
// ... other fields
@@index([transcodingStatus])
}
```
### Status Lifecycle
```
uploaded → transcoded (success)
uploaded → failed (error)
```
## Building & Deploying
### Build Docker Image
```bash
# Build locally
docker build -f Dockerfile.transcoder -t college-transcoder:latest .
# Build with custom tag
docker build -f Dockerfile.transcoder -t my-registry.com/transcoder:v1.0 .
# Push to registry
docker push my-registry.com/transcoder:v1.0
```
### Docker Compose Deployment
Update `docker-compose.yml`:
```yaml
transcoder:
image: my-registry.com/transcoder:v1.0
environment:
DATABASE_URL: postgresql://...
UPLOADS_DIR: /uploads
volumes:
- /mnt/storage/uploads:/uploads
```
### Production Checklist
- [ ] Use strong PostgreSQL password
- [ ] Set appropriate `CONCURRENT_JOBS` for your hardware
- [ ] Monitor disk space requirements
- [ ] Set up log rotation for transcoder logs
- [ ] Configure backup for HLS output directory
- [ ] Monitor transcoder health via container health checks
- [ ] Set resource limits in Docker (CPU, memory)
## Monitoring & Debugging
### View Logs
```bash
# Docker Compose
docker-compose logs -f transcoder
# Single container
docker logs -f college-transcoder
# Filter logs by level
docker logs college-transcoder | grep "\[Error\]"
```
### Check Transcoding Status
```bash
# PostgreSQL query
psql $DATABASE_URL -c \
"SELECT id, title, transcodingStatus FROM \"Video\"
ORDER BY createdAt DESC LIMIT 10;"
```
### Verify HLS Files
```bash
# Check master playlist
cat /uploads/hls/{video_id}/master.m3u8
# Verify variant playlists
for variant in 1080p 720p 480p; do
echo "=== $variant.m3u8 ===="
head -5 /uploads/hls/{video_id}/$variant.m3u8
done
# Check segment files
ls -lh /uploads/hls/{video_id}/*.ts | head -5
```
### Test with FFprobe
```bash
# Check segment file
ffprobe /uploads/hls/{video_id}/1080p_000.ts
# Check master playlist validity
ffprobe /uploads/hls/{video_id}/master.m3u8
```
## Performance Tuning
### Adjust Batch Delay
```bash
# Faster processing (shorter delay between batches)
export SLEEP_BETWEEN_BATCHES=10
# Slower processing (longer delay, less CPU usage)
export SLEEP_BETWEEN_BATCHES=60
```
### FFmpeg Presets
Edit `transcoder/index.ts` to adjust quality presets:
```typescript
const HLS_PRESETS = [
{ name: "1080p", width: 1920, height: 1080, bitrate: "3500k", maxrate: "4000k" },
{ name: "720p", width: 1280, height: 720, bitrate: "1800k", maxrate: "2000k" },
{ name: "480p", width: 854, height: 480, bitrate: "900k", maxrate: "1000k" },
];
```
## Troubleshooting
### Transcoder Won't Start
**Check logs:**
```bash
docker logs college-transcoder 2>&1 | head -50
```
**Common issues:**
- Database not accessible: Verify `DATABASE_URL`
- FFmpeg not found: Check Docker image build
- Uploads directory missing: Create `/uploads/videos` and `/uploads/hls`
### No Videos Being Transcoded
**Check video status:**
```bash
psql $DATABASE_URL -c "SELECT id, transcodingStatus FROM \"Video\";"
```
**Check file existence:**
```bash
ls -la /uploads/videos/
```
**Check permissions:**
```bash
stat /uploads/videos/ | grep Access
chmod 755 /uploads/videos/
chmod 644 /uploads/videos/*.mp4
```
### Transcoding Fails
**Check FFmpeg:**
```bash
docker exec college-transcoder ffmpeg -version
```
**Check disk space:**
```bash
df -h /uploads
# Need: ~3x original file size for temp files
```
**Check database:**
```bash
psql $DATABASE_URL
SELECT * FROM "Video" WHERE transcodingStatus = 'failed';
```
## Development
### Project Structure
```
transcoder/
├── index.ts # Main transcoder logic
├── package.json # Dependencies
├── tsconfig.json # TypeScript config
└── dist/ # Compiled output (generated)
```
### Build for Development
```bash
cd transcoder
npm install
npm run build
```
### Type Checking
```bash
cd transcoder
npx tsc --noEmit
```
## Contributing
To improve the transcoder:
1. Add new quality presets to `HLS_PRESETS`
2. Adjust FFmpeg encoding parameters
3. Add metrics/monitoring
4. Implement retry logic
5. Support additional input formats
## License
Part of the OWI CMS project.
transcoder/dist/index.d.ts
Vendored
+2
View File
@@ -0,0 +1,2 @@
export {};
//# sourceMappingURL=index.d.ts.map
transcoder/dist/index.d.ts.map
Vendored
+1
View File
@@ -0,0 +1 @@
{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../index.ts"],"names":[],"mappings":""}
transcoder/dist/index.js
Vendored
+211
View File
@@ -0,0 +1,211 @@
"use strict";
// transcoder.ts
var __createBinding = (this && this.__createBinding) || (Object.create ? (function(o, m, k, k2) {
if (k2 === undefined) k2 = k;
var desc = Object.getOwnPropertyDescriptor(m, k);
if (!desc || ("get" in desc ? !m.__esModule : desc.writable || desc.configurable)) {
desc = { enumerable: true, get: function() { return m[k]; } };
}
Object.defineProperty(o, k2, desc);
}) : (function(o, m, k, k2) {
if (k2 === undefined) k2 = k;
o[k2] = m[k];
}));
var __setModuleDefault = (this && this.__setModuleDefault) || (Object.create ? (function(o, v) {
Object.defineProperty(o, "default", { enumerable: true, value: v });
}) : function(o, v) {
o["default"] = v;
});
var __importStar = (this && this.__importStar) || (function () {
var ownKeys = function(o) {
ownKeys = Object.getOwnPropertyNames || function (o) {
var ar = [];
for (var k in o) if (Object.prototype.hasOwnProperty.call(o, k)) ar[ar.length] = k;
return ar;
};
return ownKeys(o);
};
return function (mod) {
if (mod && mod.__esModule) return mod;
var result = {};
if (mod != null) for (var k = ownKeys(mod), i = 0; i < k.length; i++) if (k[i] !== "default") __createBinding(result, mod, k[i]);
__setModuleDefault(result, mod);
return result;
};
})();
Object.defineProperty(exports, "__esModule", { value: true });
const child_process_1 = require("child_process");
const fs = __importStar(require("fs/promises"));
const path = __importStar(require("path"));
const client_1 = require("@prisma/client");
const prisma = new client_1.PrismaClient();
// -----------------------------
// Configuration
// -----------------------------
const UPLOADS_DIR = process.env.UPLOADS_DIR || "/uploads";
const ORIGINALS_DIR = path.join(UPLOADS_DIR, "videos");
const HLS_ROOT = path.join(UPLOADS_DIR, "hls");
// Safer bitrate ladder for education content
const HLS_PRESETS = [
{ name: "1080p", width: 1920, height: 1080, bitrate: "3500k", maxrate: "4000k" },
{ name: "720p", width: 1280, height: 720, bitrate: "1800k", maxrate: "2000k" },
{ name: "480p", width: 854, height: 480, bitrate: "900k", maxrate: "1000k" },
];
// -----------------------------
// Utilities
// -----------------------------
async function fileExists(p) {
try {
await fs.access(p);
return true;
}
catch {
return false;
}
}
function runFFmpeg(args) {
return new Promise((resolve, reject) => {
const ffmpeg = (0, child_process_1.spawn)("ffmpeg", args, { stdio: "inherit" });
ffmpeg.on("close", (code) => {
if (code === 0)
resolve();
else
reject(new Error(`FFmpeg exited with code ${code}`));
});
});
}
async function probeResolution(input) {
return new Promise((resolve, reject) => {
const ffprobe = (0, child_process_1.spawn)("ffprobe", [
"-v", "error",
"-select_streams", "v:0",
"-show_entries", "stream=width,height",
"-of", "csv=s=x:p=0",
input,
]);
let output = "";
ffprobe.stdout.on("data", (data) => {
output += data.toString();
});
ffprobe.on("close", (code) => {
if (code !== 0)
return reject(new Error("ffprobe failed"));
const [width, height] = output.trim().split("x").map(Number);
resolve({ width, height });
});
});
}
// -----------------------------
// HLS Creation
// -----------------------------
async function createVariant(input, outputDir, preset) {
const segmentPattern = path.join(outputDir, `${preset.name}_%03d.ts`);
const playlistPath = path.join(outputDir, `${preset.name}.m3u8`);
const args = [
"-y",
"-i", input,
"-c:v", "libx264",
"-preset", "medium",
"-profile:v", "main",
"-crf", "20",
"-vf", `scale=w=${preset.width}:h=${preset.height}:force_original_aspect_ratio=decrease`,
"-b:v", preset.bitrate,
"-maxrate", preset.maxrate,
"-bufsize", "4000k",
"-c:a", "aac",
"-b:a", "128k",
"-f", "hls",
"-hls_time", "6",
"-hls_playlist_type", "vod",
"-hls_segment_filename", segmentPattern,
playlistPath,
];
console.log(`[HLS] Creating ${preset.name}`);
await runFFmpeg(args);
}
async function createMasterPlaylist(outputDir, variants) {
const lines = [
"#EXTM3U",
"#EXT-X-VERSION:3",
];
for (const variant of variants) {
const preset = HLS_PRESETS.find(p => p.name === variant);
const bandwidth = parseInt(preset.maxrate.replace("k", "")) * 1000;
lines.push(`#EXT-X-STREAM-INF:BANDWIDTH=${bandwidth},RESOLUTION=${preset.width}x${preset.height}`);
lines.push(`${preset.name}.m3u8`);
}
await fs.writeFile(path.join(outputDir, "master.m3u8"), lines.join("\n"));
}
// -----------------------------
// Transcode One Video
// -----------------------------
async function transcodeVideo(videoId) {
console.log(`\n[Transcoding] ${videoId}`);
const inputPath = path.join(ORIGINALS_DIR, `${videoId}.mp4`);
const finalDir = path.join(HLS_ROOT, videoId);
const tempDir = path.join(HLS_ROOT, `${videoId}.tmp`);
if (!(await fileExists(inputPath))) {
throw new Error(`Original file not found: ${inputPath}`);
}
if (await fileExists(finalDir)) {
console.log(`[Skip] HLS already exists`);
return;
}
await fs.mkdir(tempDir, { recursive: true });
const { width, height } = await probeResolution(inputPath);
const allowedVariants = HLS_PRESETS.filter(p => p.width <= width && p.height <= height);
if (allowedVariants.length === 0) {
throw new Error("No suitable HLS variants for source resolution");
}
for (const preset of allowedVariants) {
await createVariant(inputPath, tempDir, preset);
}
await createMasterPlaylist(tempDir, allowedVariants.map(p => p.name));
// Atomic rename
await fs.rename(tempDir, finalDir);
console.log(`[Done] ${videoId}`);
}
// -----------------------------
// Main Batch Worker
// -----------------------------
async function main() {
console.log("[Transcoder] Starting batch run");
await fs.mkdir(HLS_ROOT, { recursive: true });
const videos = await prisma.video.findMany({
where: { transcodingStatus: "uploaded" },
});
if (videos.length === 0) {
console.log("[Transcoder] Nothing to process");
return;
}
console.log(`[Transcoder] Found ${videos.length} videos`);
for (const video of videos) {
try {
// Atomically lock job
await prisma.video.update({
where: { id: video.id },
data: { transcodingStatus: "processing" },
});
await transcodeVideo(video.id);
await prisma.video.update({
where: { id: video.id },
data: { transcodingStatus: "transcoded" },
});
}
catch (err) {
console.error(`[Error] ${video.id}`, err);
await prisma.video.update({
where: { id: video.id },
data: { transcodingStatus: "failed" },
});
}
}
await prisma.$disconnect();
console.log("[Transcoder] Batch complete");
}
main().catch(async (err) => {
console.error("[Fatal]", err);
await prisma.$disconnect();
process.exit(1);
});
//# sourceMappingURL=index.js.map
transcoder/dist/index.js.map
Vendored
+1
View File
File diff suppressed because one or more lines are too long
transcoder/index.ts
+235
View File
@@ -0,0 +1,235 @@
// transcoder.ts
import { spawn } from "child_process";
import * as fs from "fs/promises";
import * as fssync from "fs";
import * as path from "path";
import { PrismaClient } from "@prisma/client";
const prisma = new PrismaClient();
process.umask(0o022);
// -----------------------------
// Configuration
// -----------------------------
const UPLOADS_DIR = process.env.UPLOADS_DIR || "/uploads";
const ORIGINALS_DIR = path.join(UPLOADS_DIR, "videos");
const HLS_ROOT = path.join(UPLOADS_DIR, "hls");
// Safer bitrate ladder for education content
const HLS_PRESETS = [
{ name: "1080p", width: 1920, height: 1080, bitrate: "3500k", maxrate: "4000k" },
{ name: "720p", width: 1280, height: 720, bitrate: "1800k", maxrate: "2000k" },
{ name: "480p", width: 854, height: 480, bitrate: "900k", maxrate: "1000k" },
];
// -----------------------------
// Utilities
// -----------------------------
async function fileExists(p: string): Promise<boolean> {
try {
await fs.access(p);
return true;
} catch {
return false;
}
}
function runFFmpeg(args: string[]): Promise<void> {
return new Promise((resolve, reject) => {
const ffmpeg = spawn("ffmpeg", args, { stdio: "inherit" });
ffmpeg.on("close", (code) => {
if (code === 0) resolve();
else reject(new Error(`FFmpeg exited with code ${code}`));
});
});
}
async function probeResolution(input: string): Promise<{ width: number; height: number }> {
return new Promise((resolve, reject) => {
const ffprobe = spawn("ffprobe", [
"-v", "error",
"-select_streams", "v:0",
"-show_entries", "stream=width,height",
"-of", "csv=s=x:p=0",
input,
]);
let output = "";
ffprobe.stdout.on("data", (data) => {
output += data.toString();
});
ffprobe.on("close", (code) => {
if (code !== 0) return reject(new Error("ffprobe failed"));
const [width, height] = output.trim().split("x").map(Number);
resolve({ width, height });
});
});
}
// -----------------------------
// HLS Creation
// -----------------------------
async function createVariant(
input: string,
outputDir: string,
preset: typeof HLS_PRESETS[number]
) {
const segmentPattern = path.join(outputDir, `${preset.name}_%03d.ts`);
const playlistPath = path.join(outputDir, `${preset.name}.m3u8`);
const args = [
"-y",
"-i", input,
"-c:v", "libx264",
"-preset", "medium",
"-profile:v", "main",
"-crf", "20",
"-vf", `scale=w='min(${preset.width},iw)':h='min(${preset.height},ih)':force_original_aspect_ratio=decrease:force_divisible_by=2`,
"-b:v", preset.bitrate,
"-maxrate", preset.maxrate,
"-bufsize", "4000k",
"-c:a", "aac",
"-b:a", "128k",
"-f", "hls",
"-hls_time", "6",
"-hls_playlist_type", "vod",
"-hls_flags", "independent_segments",
"-hls_segment_filename", segmentPattern,
playlistPath,
];
console.log(`[HLS] Creating ${preset.name}`);
await runFFmpeg(args);
}
async function createMasterPlaylist(outputDir: string, variants: string[]): Promise<void> {
const lines: string[] = [
"#EXTM3U",
"#EXT-X-VERSION:3",
];
for (const variant of variants) {
const preset = HLS_PRESETS.find(p => p.name === variant)!;
const bandwidth = parseInt(preset.maxrate.replace("k", "")) * 1000;
lines.push(
`#EXT-X-STREAM-INF:BANDWIDTH=${bandwidth},RESOLUTION=${preset.width}x${preset.height}`
);
lines.push(`${preset.name}.m3u8`);
}
await fs.writeFile(path.join(outputDir, "master.m3u8"), lines.join("\n"));
}
// -----------------------------
// Transcode One Video
// -----------------------------
async function transcodeVideo(videoId: string) {
console.log(`\n[Transcoding] ${videoId}`);
const inputPath = path.join(ORIGINALS_DIR, `${videoId}.mp4`);
const finalDir = path.join(HLS_ROOT, videoId);
const tempDir = path.join(HLS_ROOT, `${videoId}.tmp`);
if (!(await fileExists(inputPath))) {
throw new Error(`Original file not found: ${inputPath}`);
}
if (await fileExists(finalDir)) {
console.log(`[Skip] HLS already exists`);
return;
}
await fs.mkdir(tempDir, { recursive: true });
const { width, height } = await probeResolution(inputPath);
const allowedVariants = HLS_PRESETS.filter(
p => p.width <= width && p.height <= height
);
if (allowedVariants.length === 0) {
throw new Error("No suitable HLS variants for source resolution");
}
for (const preset of allowedVariants) {
await createVariant(inputPath, tempDir, preset);
}
await createMasterPlaylist(tempDir, allowedVariants.map(p => p.name));
// Atomic rename
await fs.rename(tempDir, finalDir);
console.log(`[Done] ${videoId}`);
}
// -----------------------------
// Main Batch Worker
// -----------------------------
async function main() {
console.log("[Transcoder] Starting batch run");
await fs.mkdir(HLS_ROOT, { recursive: true });
const videos = await prisma.video.findMany({
where: { transcodingStatus: "uploaded" },
});
if (videos.length === 0) {
console.log("[Transcoder] Nothing to process");
return;
}
console.log(`[Transcoder] Found ${videos.length} videos`);
for (const video of videos) {
try {
// Atomically lock job
await prisma.video.update({
where: { id: video.id },
data: { transcodingStatus: "processing" },
});
await transcodeVideo(video.id);
await prisma.video.update({
where: { id: video.id },
data: { transcodingStatus: "transcoded" },
});
} catch (err) {
console.error(`[Error] ${video.id}`, err);
await prisma.video.update({
where: { id: video.id },
data: { transcodingStatus: "failed" },
});
}
}
await prisma.$disconnect();
console.log("[Transcoder] Batch complete");
}
main().catch(async (err) => {
console.error("[Fatal]", err);
await prisma.$disconnect();
process.exit(1);
});
transcoder/package.json
+22
View File
@@ -0,0 +1,22 @@
{
"name": "hls-transcoder",
"version": "1.0.0",
"description": "HLS transcoder service for video streaming",
"main": "dist/index.js",
"scripts": {
"start": "node dist/index.js",
"dev": "ts-node index.ts",
"build": "tsc",
"watch": "tsc --watch",
"postinstall": "prisma generate"
},
"dependencies": {
"@prisma/client": "6.19.0"
},
"devDependencies": {
"@types/node": "^20",
"prisma": "^6.19.0",
"typescript": "^5.3.3",
"ts-node": "^10.9.2"
}
}
transcoder/tsconfig.json
+19
View File
@@ -0,0 +1,19 @@
{
"compilerOptions": {
"target": "ES2020",
"module": "commonjs",
"lib": ["ES2020"],
"outDir": "./dist",
"rootDir": "./",
"strict": true,
"esModuleInterop": true,
"skipLibCheck": true,
"forceConsistentCasingInFileNames": true,
"resolveJsonModule": true,
"declaration": true,
"declarationMap": true,
"sourceMap": true
},
"include": ["index.ts"],
"exclude": ["node_modules", "dist"]
}