Vault/TRANSCODING.md

# HLS Transcoding Setup Documentation

## Overview

This system implements HLS (HTTP Live Streaming) encoding and adaptive bitrate streaming for video content. Videos are stored as MP4s initially and are automatically transcoded to HLS format using FFmpeg.

## Architecture

### Components

1. **CMS Application** (Next.js)
   - Allows admin users to upload MP4 videos
   - Videos are stored in `/uploads/videos/{video_id}.mp4`
   - Videos are marked with `transcodingStatus = 'uploaded'` initially
   - Video player supports HLS with MP4 fallback

2. **Transcoder Service** (Node.js + FFmpeg)
   - Runs as a batch job, processing all videos with `transcodingStatus = 'uploaded'`
   - Reads MP4 from storage and converts to HLS format
   - Creates multiple quality variants (based on source resolution):
     - **1080p**: 1920x1080 @ 3500 kbps (if source supports)
     - **720p**: 1280x720 @ 1800 kbps (if source supports)
     - **480p**: 854x480 @ 900 kbps (if source supports)
   - Stores files in `/uploads/hls/{video_id}.tmp/` during transcoding
   - Creates a master playlist (`master.m3u8`) combining all variants
   - Atomically renames folder from `.tmp` to final location
   - Updates database to `transcodingStatus = 'transcoded'` on success
   - Marks as `'failed'` if transcoding fails
   - Runs in a loop with configurable delay between batches

3. **Database Schema**
   - `Video` model includes `transcodingStatus` field
   - Valid values: `'uploaded'` | `'processing'` | `'transcoded'` | `'failed'`

4. **Video Player**
   - Uses `HlsPlayer` component with HLS.js
   - Automatically selects source based on availability:
     - If `transcodingStatus = 'transcoded'`: Uses `/api/videos/hls/{video_id}/master.m3u8`
     - Otherwise: Falls back to original MP4 file

## Directory Structure

```
/uploads/
├── videos/
│   ├── {video_id_1}.mp4
│   ├── {video_id_2}.mp4
│   └── ...
└── hls/
    ├── {video_id_1}/
    │   ├── master.m3u8        (Master playlist)
    │   ├── 1080p.m3u8
    │   ├── 1080p_000.ts
    │   ├── 1080p_001.ts
    │   ├── 720p.m3u8
    │   ├── 720p_000.ts
    │   ├── 480p.m3u8
    │   └── ...
    └── {video_id_2}/
        └── ...
```

## Database Migration

The migration adds `transcodingStatus` field to the `Video` model:

```sql
ALTER TABLE "Video" ADD COLUMN "transcodingStatus" TEXT NOT NULL DEFAULT 'uploaded';
CREATE INDEX "Video_transcodingStatus_idx" ON "Video"("transcodingStatus");
```

## Docker Setup

### Building the Transcoder

The transcoder is built as a multi-stage Docker image:

```bash
docker build -f Dockerfile.transcoder -t vault-transcoder:latest .
```

### Running with Docker Compose

```bash
docker-compose up -d
```

Services started:
- **cms** (port 3000): Next.js CMS application
- **postgres**: PostgreSQL database
- **transcoder**: HLS transcoding service

### Environment Variables

Create a `.env` file in the project root:

```
# Transcoder (set automatically via docker-compose)
DATABASE_URL=postgresql://cms_user:changeme@postgres:5432/cms_db
UPLOADS_DIR=/uploads
POLL_INTERVAL=5000
CONCURRENT_JOBS=2

# CMS specific (as before)
NEXTAUTH_SECRET=your-secret-key
NEXTAUTH_URL=http://localhost:3000
GOOGLE_CLIENT_ID=...
GOOGLE_CLIENT_SECRET=...
ALLOWED_ADMINS=admin@university.edu

# Paths
UPLOADS_PATH=/mnt/tank/apps/college-platform/uploads
POSTGRES_DATA_PATH=/mnt/tank/apps/college-platform/postgres_data
```

## API Endpoints

### Get Video Details

```
GET /api/videos/{videoId}
```

Response includes:

```json
{
  "video": {
    "id": "...",
    "title": "...",
    "url": "/videos/{video_id}.mp4",
    "transcodingStatus": "transcoded",
    "videoUrls": {
      "hlsUrl": "/api/videos/hls/{video_id}/master.m3u8",
      "mp4Url": "/videos/{video_id}.mp4"
    }
  }
}
```

## File Format: HLS Playlist Examples

### Master Playlist (`master.m3u8`)

```m3u8
#EXTM3U
#EXT-X-VERSION:3
#EXT-X-STREAM-INF:BANDWIDTH=3500000,RESOLUTION=1920x1080
1080p.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=1800000,RESOLUTION=1280x720
720p.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=900000,RESOLUTION=854x480
480p.m3u8
```

### Variant Playlist (`1080p.m3u8`)

```m3u8
#EXTM3U
#EXT-X-VERSION:3
#EXT-X-TARGETDURATION:6
#EXT-X-PLAYLIST-TYPE:VOD
#EXTINF:6.0,
1080p_000.ts
#EXTINF:6.0,
1080p_001.ts
...
#EXT-X-ENDLIST
```

## Development Mode

For local development without Docker:

1. Ensure FFmpeg is installed on your system
2. Set environment variables:
   ```bash
   export DATABASE_URL=postgresql://cms_user:changeme@localhost:5432/cms_db
   export UPLOADS_DIR=/path/to/uploads
   ```
3. Run the transcoder:
   ```bash
   cd transcoder
   npm install
   npm run dev
   ```

## Monitoring & Logs

### CMS Logs

```bash
docker logs college-cms
```

### Transcoder Logs

```bash
docker logs college-transcoder
```

Look for lines starting with:
- `[Transcoding]` - Transcoding progress
- `[HLS]` - HLS specific operations
- `[Database]` - Database updates
- `[Error]` - Any errors encountered

### Checking Transcoding Status

```bash
# Check video transcoding status
psql -U cms_user -d cms_db -c "SELECT id, title, transcodingStatus FROM \"Video\";"

# Monitor real-time transcoding
watch -n 1 'docker exec college-transcoder head -20 /app/transcoder.log'
```

## Troubleshooting

### Transcoder Not Finding Videos

**Issue**: Transcoder logs show "MP4 file not found"

**Solution**:
1. Verify MP4 files are in `/uploads/videos/` directory
2. Check file permissions: `chmod 644 /uploads/videos/*.mp4`
3. Ensure the path matches the `UPLOADS_DIR` environment variable

### HLS Files Not Generated

**Issue**: Videos stay in `'uploaded'` state

**Solution**:
1. Check FFmpeg is installed in container: `docker exec college-transcoder ffmpeg -version`
2. Check database connectivity: `docker logs college-transcoder | grep "DATABASE_URL"`
3. Verify disk space: `df -h /uploads`

### Video Player Shows MP4 Instead of HLS

**Issue**: HLS not playing even after transcoding

**Solution**:
1. Verify `transcodingStatus = 'transcoded'` in database
2. Check HLS files exist: `ls /uploads/hls/{video_id}/`
3. Verify master playlist is valid: `cat /uploads/hls/{video_id}/master.m3u8`

### Performance Issues

**Recommendation**: Adjust `CONCURRENT_JOBS` based on available system resources:
- 1-2 jobs for systems with <4 CPU cores
- 2-4 jobs for systems with 4-8 CPU cores
- Increase `POLL_INTERVAL` if CPU usage is high

## Client-Side Configuration

The video player is configured to:

1. Try HLS playlist first if available
2. Fall back to MP4 on HLS failure
3. Use adaptive bitrate selection when HLS is available
4. Support all modern browsers (Chrome, Firefox, Safari, Edge)

## Security Considerations

1. **Storage Access**: Only the transcoder container has write access to `/uploads/hls/`
2. **Database**: Use strong PostgreSQL password in production
3. **Network**: Both containers connect through Docker network, no external ports exposed
4. **File Validation**: Consider adding video file validation before transcoding

## Future Enhancements

1. Add thumbnail extraction during transcoding
2. Implement retry logic for failed transcoding jobs
3. Add progress tracking API for long-running transcodes
4. Support multiple audio tracks/subtitles
5. Implement cache invalidation for CDN
6. Add metrics and monitoring dashboard