Vault/transcoder/README.md

# HLS Transcoder Service

A scalable HLS (HTTP Live Streaming) transcoding service for converting MP4 videos into adaptive bitrate HLS streams using FFmpeg.

## Features

- ✅ Automatic video detection and transcoding
- ✅ Multiple quality variants (1080p, 720p, 480p)
- ✅ Master playlist generation for adaptive bitrate streaming
- ✅ Concurrent job processing
- ✅ Database integration with Prisma
- ✅ Comprehensive error handling and logging
- ✅ Docker support with multi-stage builds
- ✅ Health checks and graceful shutdown

## Prerequisites

### For Local Development
- Node.js 18+
- FFmpeg
- PostgreSQL
- TypeScript

### For Docker
- Docker Desktop (with Windows Support)
- Docker Compose 2.0+

## Quick Start

### Development Mode

```bash
# Install dependencies
cd transcoder
npm install

# Set environment variables
export DATABASE_URL="postgresql://cms_user:changeme@localhost:5432/cms_db"
export UPLOADS_DIR="/path/to/uploads"
export NODE_ENV="development"

# Run in development mode
npm run dev

# Or build and run
npm run build
npm start
```

### Docker Compose

```bash
# From project root
docker-compose up -d

# View logs
docker-compose logs -f transcoder

# Stop services
docker-compose down
```

## Configuration

### Environment Variables

| Variable | Description | Default |
|----------|-------------|---------|
| `DATABASE_URL` | PostgreSQL connection string | (required) |
| `UPLOADS_DIR` | Path to uploads directory | `/uploads` |
| `POLL_INTERVAL` | Check for new videos every N ms | `5000` |
| `CONCURRENT_JOBS` | Videos to transcode simultaneously | `2` |
| `NODE_ENV` | Environment (development/production) | `production` |

### Example Configuration

```bash
DATABASE_URL=postgresql://cms_user:secure_password@db.example.com:5432/cms_db
UPLOADS_DIR=/mnt/storage/uploads
POLL_INTERVAL=10000
CONCURRENT_JOBS=4
```

## Output Format

### Directory Structure
```
/uploads/hls/
├── {video_id_1}/
│   ├── master.m3u8      # Master playlist
│   ├── 1080p.m3u8       # 1080p variant
│   ├── 1080p_000.ts     # TS segments
│   ├── 1080p_001.ts
│   ├── 720p.m3u8        # 720p variant
│   ├── 720p_000.ts
│   └── 480p.m3u8        # 480p variant
└── {video_id_2}/
    └── ...
```

### Master Playlist Format

```m3u8
#EXTM3U
#EXT-X-VERSION:3
#EXT-X-STREAM-INF:BANDWIDTH=3500000,RESOLUTION=1920x1080
1080p.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=1800000,RESOLUTION=1280x720
720p.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=900000,RESOLUTION=854x480
480p.m3u8
```

## API Reference

### Database Schema

```prisma
model Video {
  id                  String   @id
  transcodingStatus   String   @default("uploaded")
  // ... other fields
  
  @@index([transcodingStatus])
}
```

### Status Lifecycle

```
uploaded → transcoded  (success)
uploaded → failed      (error)
```

## Building & Deploying

### Build Docker Image

```bash
# Build locally
docker build -f Dockerfile.transcoder -t college-transcoder:latest .

# Build with custom tag
docker build -f Dockerfile.transcoder -t my-registry.com/transcoder:v1.0 .

# Push to registry
docker push my-registry.com/transcoder:v1.0
```

### Docker Compose Deployment

Update `docker-compose.yml`:

```yaml
transcoder:
  image: my-registry.com/transcoder:v1.0
  environment:
    DATABASE_URL: postgresql://...
    UPLOADS_DIR: /uploads
  volumes:
    - /mnt/storage/uploads:/uploads
```

### Production Checklist

- [ ] Use strong PostgreSQL password
- [ ] Set appropriate `CONCURRENT_JOBS` for your hardware
- [ ] Monitor disk space requirements
- [ ] Set up log rotation for transcoder logs
- [ ] Configure backup for HLS output directory
- [ ] Monitor transcoder health via container health checks
- [ ] Set resource limits in Docker (CPU, memory)

## Monitoring & Debugging

### View Logs

```bash
# Docker Compose
docker-compose logs -f transcoder

# Single container
docker logs -f college-transcoder

# Filter logs by level
docker logs college-transcoder | grep "\[Error\]"
```

### Check Transcoding Status

```bash
# PostgreSQL query
psql $DATABASE_URL -c \
  "SELECT id, title, transcodingStatus FROM \"Video\" 
   ORDER BY createdAt DESC LIMIT 10;"
```

### Verify HLS Files

```bash
# Check master playlist
cat /uploads/hls/{video_id}/master.m3u8

# Verify variant playlists
for variant in 1080p 720p 480p; do
  echo "=== $variant.m3u8 ===="
  head -5 /uploads/hls/{video_id}/$variant.m3u8
done

# Check segment files
ls -lh /uploads/hls/{video_id}/*.ts | head -5
```

### Test with FFprobe

```bash
# Check segment file
ffprobe /uploads/hls/{video_id}/1080p_000.ts

# Check master playlist validity
ffprobe /uploads/hls/{video_id}/master.m3u8
```

## Performance Tuning

### Adjust Batch Delay

```bash
# Faster processing (shorter delay between batches)
export SLEEP_BETWEEN_BATCHES=10

# Slower processing (longer delay, less CPU usage)
export SLEEP_BETWEEN_BATCHES=60
```

### FFmpeg Presets

Edit `transcoder/index.ts` to adjust quality presets:

```typescript
const HLS_PRESETS = [
  { name: "1080p", width: 1920, height: 1080, bitrate: "3500k", maxrate: "4000k" },
  { name: "720p", width: 1280, height: 720, bitrate: "1800k", maxrate: "2000k" },
  { name: "480p", width: 854, height: 480, bitrate: "900k", maxrate: "1000k" },
];
```

## Troubleshooting

### Transcoder Won't Start

**Check logs:**
```bash
docker logs college-transcoder 2>&1 | head -50
```

**Common issues:**
- Database not accessible: Verify `DATABASE_URL`
- FFmpeg not found: Check Docker image build
- Uploads directory missing: Create `/uploads/videos` and `/uploads/hls`

### No Videos Being Transcoded

**Check video status:**
```bash
psql $DATABASE_URL -c "SELECT id, transcodingStatus FROM \"Video\";"
```

**Check file existence:**
```bash
ls -la /uploads/videos/
```

**Check permissions:**
```bash
stat /uploads/videos/ | grep Access
chmod 755 /uploads/videos/
chmod 644 /uploads/videos/*.mp4
```

### Transcoding Fails

**Check FFmpeg:**
```bash
docker exec college-transcoder ffmpeg -version
```

**Check disk space:**
```bash
df -h /uploads
# Need: ~3x original file size for temp files
```

**Check database:**
```bash
psql $DATABASE_URL
SELECT * FROM "Video" WHERE transcodingStatus = 'failed';
```

## Development

### Project Structure

```
transcoder/
├── index.ts           # Main transcoder logic
├── package.json       # Dependencies
├── tsconfig.json      # TypeScript config
└── dist/              # Compiled output (generated)
```

### Build for Development

```bash
cd transcoder
npm install
npm run build
```

### Type Checking

```bash
cd transcoder
npx tsc --noEmit
```

## Contributing

To improve the transcoder:

1. Add new quality presets to `HLS_PRESETS`
2. Adjust FFmpeg encoding parameters
3. Add metrics/monitoring
4. Implement retry logic
5. Support additional input formats

## License

Part of the OWI CMS project.