Vault/transcoder/README.md

HLS Transcoder Service

A scalable HLS (HTTP Live Streaming) transcoding service for converting MP4 videos into adaptive bitrate HLS streams using FFmpeg.

Features

• ✅ Automatic video detection and transcoding
• ✅ Multiple quality variants (1080p, 720p, 480p)
• ✅ Master playlist generation for adaptive bitrate streaming
• ✅ Concurrent job processing
• ✅ Database integration with Prisma
• ✅ Comprehensive error handling and logging
• ✅ Docker support with multi-stage builds
• ✅ Health checks and graceful shutdown

Prerequisites

For Local Development

• Node.js 18+
• FFmpeg
• PostgreSQL
• TypeScript

For Docker

• Docker Desktop (with Windows Support)
• Docker Compose 2.0+

Quick Start

Development Mode

# Install dependencies
cd transcoder
npm install

# Set environment variables
export DATABASE_URL="postgresql://cms_user:changeme@localhost:5432/cms_db"
export UPLOADS_DIR="/path/to/uploads"
export NODE_ENV="development"

# Run in development mode
npm run dev

# Or build and run
npm run build
npm start

Docker Compose

# From project root
docker-compose up -d

# View logs
docker-compose logs -f transcoder

# Stop services
docker-compose down

Configuration

Environment Variables

Variable	Description	Default
DATABASE_URL	PostgreSQL connection string	(required)
UPLOADS_DIR	Path to uploads directory	/uploads
POLL_INTERVAL	Check for new videos every N ms	5000
CONCURRENT_JOBS	Videos to transcode simultaneously	2
NODE_ENV	Environment (development/production)	production

Example Configuration

DATABASE_URL=postgresql://cms_user:secure_password@db.example.com:5432/cms_db
UPLOADS_DIR=/mnt/storage/uploads
POLL_INTERVAL=10000
CONCURRENT_JOBS=4

Output Format

Directory Structure

/uploads/hls/
├── {video_id_1}/
│   ├── master.m3u8      # Master playlist
│   ├── 1080p.m3u8       # 1080p variant
│   ├── 1080p_000.ts     # TS segments
│   ├── 1080p_001.ts
│   ├── 720p.m3u8        # 720p variant
│   ├── 720p_000.ts
│   └── 480p.m3u8        # 480p variant
└── {video_id_2}/
    └── ...

Master Playlist Format

#EXTM3U
#EXT-X-VERSION:3
#EXT-X-STREAM-INF:BANDWIDTH=3500000,RESOLUTION=1920x1080
1080p.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=1800000,RESOLUTION=1280x720
720p.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=900000,RESOLUTION=854x480
480p.m3u8

API Reference

Database Schema

model Video {
  id                  String   @id
  transcodingStatus   String   @default("uploaded")
  // ... other fields
  
  @@index([transcodingStatus])
}

Status Lifecycle

uploaded → transcoded  (success)
uploaded → failed      (error)

Building & Deploying

Build Docker Image

# Build locally
docker build -f Dockerfile.transcoder -t college-transcoder:latest .

# Build with custom tag
docker build -f Dockerfile.transcoder -t my-registry.com/transcoder:v1.0 .

# Push to registry
docker push my-registry.com/transcoder:v1.0

Docker Compose Deployment

Update docker-compose.yml:

transcoder:
  image: my-registry.com/transcoder:v1.0
  environment:
    DATABASE_URL: postgresql://...
    UPLOADS_DIR: /uploads
  volumes:
    - /mnt/storage/uploads:/uploads

Production Checklist

• Use strong PostgreSQL password
• Set appropriate CONCURRENT_JOBS for your hardware
• Monitor disk space requirements
• Set up log rotation for transcoder logs
• Configure backup for HLS output directory
• Monitor transcoder health via container health checks
• Set resource limits in Docker (CPU, memory)

Monitoring & Debugging

View Logs

# Docker Compose
docker-compose logs -f transcoder

# Single container
docker logs -f college-transcoder

# Filter logs by level
docker logs college-transcoder | grep "\[Error\]"

Check Transcoding Status

# PostgreSQL query
psql $DATABASE_URL -c \
  "SELECT id, title, transcodingStatus FROM \"Video\" 
   ORDER BY createdAt DESC LIMIT 10;"

Verify HLS Files

# Check master playlist
cat /uploads/hls/{video_id}/master.m3u8

# Verify variant playlists
for variant in 1080p 720p 480p; do
  echo "=== $variant.m3u8 ===="
  head -5 /uploads/hls/{video_id}/$variant.m3u8
done

# Check segment files
ls -lh /uploads/hls/{video_id}/*.ts | head -5

Test with FFprobe

# Check segment file
ffprobe /uploads/hls/{video_id}/1080p_000.ts

# Check master playlist validity
ffprobe /uploads/hls/{video_id}/master.m3u8

Performance Tuning

Adjust Batch Delay

# Faster processing (shorter delay between batches)
export SLEEP_BETWEEN_BATCHES=10

# Slower processing (longer delay, less CPU usage)
export SLEEP_BETWEEN_BATCHES=60

FFmpeg Presets

Edit transcoder/index.ts to adjust quality presets:

const HLS_PRESETS = [
  { name: "1080p", width: 1920, height: 1080, bitrate: "3500k", maxrate: "4000k" },
  { name: "720p", width: 1280, height: 720, bitrate: "1800k", maxrate: "2000k" },
  { name: "480p", width: 854, height: 480, bitrate: "900k", maxrate: "1000k" },
];

Troubleshooting

Transcoder Won't Start

Check logs:

docker logs college-transcoder 2>&1 | head -50

Common issues:

• Database not accessible: Verify DATABASE_URL
• FFmpeg not found: Check Docker image build
• Uploads directory missing: Create /uploads/videos and /uploads/hls

No Videos Being Transcoded

Check video status:

psql $DATABASE_URL -c "SELECT id, transcodingStatus FROM \"Video\";"

Check file existence:

ls -la /uploads/videos/

Check permissions:

stat /uploads/videos/ | grep Access
chmod 755 /uploads/videos/
chmod 644 /uploads/videos/*.mp4

Transcoding Fails

Check FFmpeg:

docker exec college-transcoder ffmpeg -version

Check disk space:

df -h /uploads
# Need: ~3x original file size for temp files

Check database:

psql $DATABASE_URL
SELECT * FROM "Video" WHERE transcodingStatus = 'failed';

Development

Project Structure

transcoder/
├── index.ts           # Main transcoder logic
├── package.json       # Dependencies
├── tsconfig.json      # TypeScript config
└── dist/              # Compiled output (generated)

Build for Development

cd transcoder
npm install
npm run build

Type Checking

cd transcoder
npx tsc --noEmit

Contributing

To improve the transcoder:

1. Add new quality presets to HLS_PRESETS
2. Adjust FFmpeg encoding parameters
3. Add metrics/monitoring
4. Implement retry logic
5. Support additional input formats

License

Part of the OWI CMS project.