Vault/TRANSCODING.md

HLS Transcoding Setup Documentation

Overview

This system implements HLS (HTTP Live Streaming) encoding and adaptive bitrate streaming for video content. Videos are stored as MP4s initially and are automatically transcoded to HLS format using FFmpeg.

Architecture

Components

1. CMS Application (Next.js)

   • Allows admin users to upload MP4 videos
   • Videos are stored in /uploads/videos/{video_id}.mp4
   • Videos are marked with transcodingStatus = 'uploaded' initially
   • Video player supports HLS with MP4 fallback

2. Transcoder Service (Node.js + FFmpeg)

   • Runs as a batch job, processing all videos with transcodingStatus = 'uploaded'
   • Reads MP4 from storage and converts to HLS format
   • Creates multiple quality variants (based on source resolution):
     • 1080p: 1920x1080 @ 3500 kbps (if source supports)
     • 720p: 1280x720 @ 1800 kbps (if source supports)
     • 480p: 854x480 @ 900 kbps (if source supports)
   • Stores files in /uploads/hls/{video_id}.tmp/ during transcoding
   • Creates a master playlist (master.m3u8) combining all variants
   • Atomically renames folder from .tmp to final location
   • Updates database to transcodingStatus = 'transcoded' on success
   • Marks as 'failed' if transcoding fails
   • Runs in a loop with configurable delay between batches

3. Database Schema

   • Video model includes transcodingStatus field
   • Valid values: 'uploaded' | 'processing' | 'transcoded' | 'failed'

4. Video Player

   • Uses HlsPlayer component with HLS.js
   • Automatically selects source based on availability:
     • If transcodingStatus = 'transcoded': Uses /api/videos/hls/{video_id}/master.m3u8
     • Otherwise: Falls back to original MP4 file

Directory Structure

/uploads/
├── videos/
│   ├── {video_id_1}.mp4
│   ├── {video_id_2}.mp4
│   └── ...
└── hls/
    ├── {video_id_1}/
    │   ├── master.m3u8        (Master playlist)
    │   ├── 1080p.m3u8
    │   ├── 1080p_000.ts
    │   ├── 1080p_001.ts
    │   ├── 720p.m3u8
    │   ├── 720p_000.ts
    │   ├── 480p.m3u8
    │   └── ...
    └── {video_id_2}/
        └── ...

Database Migration

The migration adds transcodingStatus field to the Video model:

ALTER TABLE "Video" ADD COLUMN "transcodingStatus" TEXT NOT NULL DEFAULT 'uploaded';
CREATE INDEX "Video_transcodingStatus_idx" ON "Video"("transcodingStatus");

Docker Setup

Building the Transcoder

The transcoder is built as a multi-stage Docker image:

docker build -f Dockerfile.transcoder -t vault-transcoder:latest .

Running with Docker Compose

docker-compose up -d

Services started:

• cms (port 3000): Next.js CMS application
• postgres: PostgreSQL database
• transcoder: HLS transcoding service

Environment Variables

Create a .env file in the project root:

# Transcoder (set automatically via docker-compose)
DATABASE_URL=postgresql://cms_user:changeme@postgres:5432/cms_db
UPLOADS_DIR=/uploads
POLL_INTERVAL=5000
CONCURRENT_JOBS=2

# CMS specific (as before)
NEXTAUTH_SECRET=your-secret-key
NEXTAUTH_URL=http://localhost:3000
GOOGLE_CLIENT_ID=...
GOOGLE_CLIENT_SECRET=...
ALLOWED_ADMINS=admin@university.edu

# Paths
UPLOADS_PATH=/mnt/tank/apps/college-platform/uploads
POSTGRES_DATA_PATH=/mnt/tank/apps/college-platform/postgres_data

API Endpoints

Get Video Details

GET /api/videos/{videoId}

Response includes:

{
  "video": {
    "id": "...",
    "title": "...",
    "url": "/videos/{video_id}.mp4",
    "transcodingStatus": "transcoded",
    "videoUrls": {
      "hlsUrl": "/api/videos/hls/{video_id}/master.m3u8",
      "mp4Url": "/videos/{video_id}.mp4"
    }
  }
}

File Format: HLS Playlist Examples

Master Playlist (master.m3u8)

#EXTM3U
#EXT-X-VERSION:3
#EXT-X-STREAM-INF:BANDWIDTH=3500000,RESOLUTION=1920x1080
1080p.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=1800000,RESOLUTION=1280x720
720p.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=900000,RESOLUTION=854x480
480p.m3u8

Variant Playlist (1080p.m3u8)

#EXTM3U
#EXT-X-VERSION:3
#EXT-X-TARGETDURATION:6
#EXT-X-PLAYLIST-TYPE:VOD
#EXTINF:6.0,
1080p_000.ts
#EXTINF:6.0,
1080p_001.ts
...
#EXT-X-ENDLIST

Development Mode

For local development without Docker:

1. Ensure FFmpeg is installed on your system
2. Set environment variables:

   export DATABASE_URL=postgresql://cms_user:changeme@localhost:5432/cms_db
   export UPLOADS_DIR=/path/to/uploads
   

3. Run the transcoder:

   cd transcoder
   npm install
   npm run dev
   

Monitoring & Logs

CMS Logs

docker logs college-cms

Transcoder Logs

docker logs college-transcoder

Look for lines starting with:

• [Transcoding] - Transcoding progress
• [HLS] - HLS specific operations
• [Database] - Database updates
• [Error] - Any errors encountered

Checking Transcoding Status

# Check video transcoding status
psql -U cms_user -d cms_db -c "SELECT id, title, transcodingStatus FROM \"Video\";"

# Monitor real-time transcoding
watch -n 1 'docker exec college-transcoder head -20 /app/transcoder.log'

Troubleshooting

Transcoder Not Finding Videos

Issue: Transcoder logs show "MP4 file not found"

Solution:

1. Verify MP4 files are in /uploads/videos/ directory
2. Check file permissions: chmod 644 /uploads/videos/*.mp4
3. Ensure the path matches the UPLOADS_DIR environment variable

HLS Files Not Generated

Issue: Videos stay in 'uploaded' state

Solution:

1. Check FFmpeg is installed in container: docker exec college-transcoder ffmpeg -version
2. Check database connectivity: docker logs college-transcoder | grep "DATABASE_URL"
3. Verify disk space: df -h /uploads

Video Player Shows MP4 Instead of HLS

Issue: HLS not playing even after transcoding

Solution:

1. Verify transcodingStatus = 'transcoded' in database
2. Check HLS files exist: ls /uploads/hls/{video_id}/
3. Verify master playlist is valid: cat /uploads/hls/{video_id}/master.m3u8

Performance Issues

Recommendation: Adjust CONCURRENT_JOBS based on available system resources:

• 1-2 jobs for systems with <4 CPU cores
• 2-4 jobs for systems with 4-8 CPU cores
• Increase POLL_INTERVAL if CPU usage is high

Client-Side Configuration

The video player is configured to:

1. Try HLS playlist first if available
2. Fall back to MP4 on HLS failure
3. Use adaptive bitrate selection when HLS is available
4. Support all modern browsers (Chrome, Firefox, Safari, Edge)

Security Considerations

1. Storage Access: Only the transcoder container has write access to /uploads/hls/
2. Database: Use strong PostgreSQL password in production
3. Network: Both containers connect through Docker network, no external ports exposed
4. File Validation: Consider adding video file validation before transcoding

Future Enhancements

1. Add thumbnail extraction during transcoding
2. Implement retry logic for failed transcoding jobs
3. Add progress tracking API for long-running transcodes
4. Support multiple audio tracks/subtitles
5. Implement cache invalidation for CDN
6. Add metrics and monitoring dashboard