Initial commit

2026-06-11 10:46:09 +02:00
commit 81ad7e4ea9
223 changed files with 39530 additions and 0 deletions
@@ -0,0 +1,277 @@
+# HLS Transcoding Setup Documentation
+
+## Overview
+
+This system implements HLS (HTTP Live Streaming) encoding and adaptive bitrate streaming for video content. Videos are stored as MP4s initially and are automatically transcoded to HLS format using FFmpeg.
+
+## Architecture
+
+### Components
+
+1. **CMS Application** (Next.js)
+   - Allows admin users to upload MP4 videos
+   - Videos are stored in `/uploads/videos/{video_id}.mp4`
+   - Videos are marked with `transcodingStatus = 'uploaded'` initially
+   - Video player supports HLS with MP4 fallback
+
+2. **Transcoder Service** (Node.js + FFmpeg)
+   - Runs as a batch job, processing all videos with `transcodingStatus = 'uploaded'`
+   - Reads MP4 from storage and converts to HLS format
+   - Creates multiple quality variants (based on source resolution):
+     - **1080p**: 1920x1080 @ 3500 kbps (if source supports)
+     - **720p**: 1280x720 @ 1800 kbps (if source supports)
+     - **480p**: 854x480 @ 900 kbps (if source supports)
+   - Stores files in `/uploads/hls/{video_id}.tmp/` during transcoding
+   - Creates a master playlist (`master.m3u8`) combining all variants
+   - Atomically renames folder from `.tmp` to final location
+   - Updates database to `transcodingStatus = 'transcoded'` on success
+   - Marks as `'failed'` if transcoding fails
+   - Runs in a loop with configurable delay between batches
+
+3. **Database Schema**
+   - `Video` model includes `transcodingStatus` field
+   - Valid values: `'uploaded'` | `'processing'` | `'transcoded'` | `'failed'`
+
+4. **Video Player**
+   - Uses `HlsPlayer` component with HLS.js
+   - Automatically selects source based on availability:
+     - If `transcodingStatus = 'transcoded'`: Uses `/api/videos/hls/{video_id}/master.m3u8`
+     - Otherwise: Falls back to original MP4 file
+
+## Directory Structure
+
+```
+/uploads/
+├── videos/
+│   ├── {video_id_1}.mp4
+│   ├── {video_id_2}.mp4
+│   └── ...
+└── hls/
+    ├── {video_id_1}/
+    │   ├── master.m3u8        (Master playlist)
+    │   ├── 1080p.m3u8
+    │   ├── 1080p_000.ts
+    │   ├── 1080p_001.ts
+    │   ├── 720p.m3u8
+    │   ├── 720p_000.ts
+    │   ├── 480p.m3u8
+    │   └── ...
+    └── {video_id_2}/
+        └── ...
+```
+
+## Database Migration
+
+The migration adds `transcodingStatus` field to the `Video` model:
+
+```sql
+ALTER TABLE "Video" ADD COLUMN "transcodingStatus" TEXT NOT NULL DEFAULT 'uploaded';
+CREATE INDEX "Video_transcodingStatus_idx" ON "Video"("transcodingStatus");
+```
+
+## Docker Setup
+
+### Building the Transcoder
+
+The transcoder is built as a multi-stage Docker image:
+
+```bash
+docker build -f Dockerfile.transcoder -t vault-transcoder:latest .
+```
+
+### Running with Docker Compose
+
+```bash
+docker-compose up -d
+```
+
+Services started:
+- **cms** (port 3000): Next.js CMS application
+- **postgres**: PostgreSQL database
+- **transcoder**: HLS transcoding service
+
+### Environment Variables
+
+Create a `.env` file in the project root:
+
+```
+# Transcoder (set automatically via docker-compose)
+DATABASE_URL=postgresql://cms_user:changeme@postgres:5432/cms_db
+UPLOADS_DIR=/uploads
+POLL_INTERVAL=5000
+CONCURRENT_JOBS=2
+
+# CMS specific (as before)
+NEXTAUTH_SECRET=your-secret-key
+NEXTAUTH_URL=http://localhost:3000
+GOOGLE_CLIENT_ID=...
+GOOGLE_CLIENT_SECRET=...
+ALLOWED_ADMINS=admin@university.edu
+
+# Paths
+UPLOADS_PATH=/mnt/tank/apps/college-platform/uploads
+POSTGRES_DATA_PATH=/mnt/tank/apps/college-platform/postgres_data
+```
+
+## API Endpoints
+
+### Get Video Details
+
+```
+GET /api/videos/{videoId}
+```
+
+Response includes:
+
+```json
+{
+  "video": {
+    "id": "...",
+    "title": "...",
+    "url": "/videos/{video_id}.mp4",
+    "transcodingStatus": "transcoded",
+    "videoUrls": {
+      "hlsUrl": "/api/videos/hls/{video_id}/master.m3u8",
+      "mp4Url": "/videos/{video_id}.mp4"
+    }
+  }
+}
+```
+
+## File Format: HLS Playlist Examples
+
+### Master Playlist (`master.m3u8`)
+
+```m3u8
+#EXTM3U
+#EXT-X-VERSION:3
+#EXT-X-STREAM-INF:BANDWIDTH=3500000,RESOLUTION=1920x1080
+1080p.m3u8
+#EXT-X-STREAM-INF:BANDWIDTH=1800000,RESOLUTION=1280x720
+720p.m3u8
+#EXT-X-STREAM-INF:BANDWIDTH=900000,RESOLUTION=854x480
+480p.m3u8
+```
+
+### Variant Playlist (`1080p.m3u8`)
+
+```m3u8
+#EXTM3U
+#EXT-X-VERSION:3
+#EXT-X-TARGETDURATION:6
+#EXT-X-PLAYLIST-TYPE:VOD
+#EXTINF:6.0,
+1080p_000.ts
+#EXTINF:6.0,
+1080p_001.ts
+...
+#EXT-X-ENDLIST
+```
+
+## Development Mode
+
+For local development without Docker:
+
+1. Ensure FFmpeg is installed on your system
+2. Set environment variables:
+   ```bash
+   export DATABASE_URL=postgresql://cms_user:changeme@localhost:5432/cms_db
+   export UPLOADS_DIR=/path/to/uploads
+   ```
+3. Run the transcoder:
+   ```bash
+   cd transcoder
+   npm install
+   npm run dev
+   ```
+
+## Monitoring & Logs
+
+### CMS Logs
+
+```bash
+docker logs college-cms
+```
+
+### Transcoder Logs
+
+```bash
+docker logs college-transcoder
+```
+
+Look for lines starting with:
+- `[Transcoding]` - Transcoding progress
+- `[HLS]` - HLS specific operations
+- `[Database]` - Database updates
+- `[Error]` - Any errors encountered
+
+### Checking Transcoding Status
+
+```bash
+# Check video transcoding status
+psql -U cms_user -d cms_db -c "SELECT id, title, transcodingStatus FROM \"Video\";"
+
+# Monitor real-time transcoding
+watch -n 1 'docker exec college-transcoder head -20 /app/transcoder.log'
+```
+
+## Troubleshooting
+
+### Transcoder Not Finding Videos
+
+**Issue**: Transcoder logs show "MP4 file not found"
+
+**Solution**:
+1. Verify MP4 files are in `/uploads/videos/` directory
+2. Check file permissions: `chmod 644 /uploads/videos/*.mp4`
+3. Ensure the path matches the `UPLOADS_DIR` environment variable
+
+### HLS Files Not Generated
+
+**Issue**: Videos stay in `'uploaded'` state
+
+**Solution**:
+1. Check FFmpeg is installed in container: `docker exec college-transcoder ffmpeg -version`
+2. Check database connectivity: `docker logs college-transcoder | grep "DATABASE_URL"`
+3. Verify disk space: `df -h /uploads`
+
+### Video Player Shows MP4 Instead of HLS
+
+**Issue**: HLS not playing even after transcoding
+
+**Solution**:
+1. Verify `transcodingStatus = 'transcoded'` in database
+2. Check HLS files exist: `ls /uploads/hls/{video_id}/`
+3. Verify master playlist is valid: `cat /uploads/hls/{video_id}/master.m3u8`
+
+### Performance Issues
+
+**Recommendation**: Adjust `CONCURRENT_JOBS` based on available system resources:
+- 1-2 jobs for systems with <4 CPU cores
+- 2-4 jobs for systems with 4-8 CPU cores
+- Increase `POLL_INTERVAL` if CPU usage is high
+
+## Client-Side Configuration
+
+The video player is configured to:
+
+1. Try HLS playlist first if available
+2. Fall back to MP4 on HLS failure
+3. Use adaptive bitrate selection when HLS is available
+4. Support all modern browsers (Chrome, Firefox, Safari, Edge)
+
+## Security Considerations
+
+1. **Storage Access**: Only the transcoder container has write access to `/uploads/hls/`
+2. **Database**: Use strong PostgreSQL password in production
+3. **Network**: Both containers connect through Docker network, no external ports exposed
+4. **File Validation**: Consider adding video file validation before transcoding
+
+## Future Enhancements
+
+1. Add thumbnail extraction during transcoding
+2. Implement retry logic for failed transcoding jobs
+3. Add progress tracking API for long-running transcodes
+4. Support multiple audio tracks/subtitles
+5. Implement cache invalidation for CDN
+6. Add metrics and monitoring dashboard