Files

T

twotalesanimation 4cd8986e69 Stream video uploads to disk instead of buffering in memory

req.formData() parsed the entire multipart body into RAM, so large
uploads (1.6GB+) OOM-killed the server. Bridge the web Request body to
a Node stream and parse it with formidable, which writes the file to
disk as it arrives:

- Temp dir is /uploads/tmp (same filesystem as the destination) so the
  post-parse move is an atomic rename, not a second multi-GB write
- Mark the bridged request as chunked when a proxy strips
  content-length, otherwise formidable assumes an empty body
- Track formidable temp files via fileBegin so aborted uploads get
  their partial files cleaned up
- Remove the dead formidable branch gated on (req as any).req, which
  is always null in the App Router

Also fix the nginx example in VPS_UPLOAD_CONFIG.md: client_max_body_size
was 1024M, below the app's 2.5GB limit, and would 413 large uploads at
the proxy before they reached the app.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

2026-08-01 15:54:57 +02:00

3.1 KiB

Raw Blame History

VPS Upload Timeout Fix

Problem

Uploads timing out after moving to VPS + npm environment.

Root Causes

Prisma connection pool timeout too short for network latency
FFprobe duration extraction timing out on slow disk I/O
Missing socket/request timeout configuration
Database connection pool exhaustion

Solutions Applied

1. Prisma Client Configuration

Updated lib/prisma.ts with:

Connection timeout handling for slow VPS networks
Proper pool configuration support
Removed 'info' logging (reduces noise in production)

2. FFprobe Timeout Protection

Updated app/api/admin/upload/route.ts:

Added 15-second timeout for FFprobe duration extraction
Falls back gracefully if FFprobe takes too long
Prevents blocking the entire upload handler

3. Next.js Server Configuration

Updated next.config.ts:

Added socket timeout configuration (45 seconds default)
VPS-aware settings for slow network environments

Required Environment Variables

Add these to your .env.local (or VPS environment):

# Database connection with proper pool settings for VPS
# Adjust pool_max based on your VPS CPU cores
DATABASE_URL="postgresql://user:password@host:5432/dbname?schema=public&pool_max=5&socket_timeout=45000&connect_timeout=10000"

# Optional: Override default socket timeout (in milliseconds)
DATABASE_SOCKET_TIMEOUT=45000

# Optional: For testing, you can increase the API timeout
# This is already set to 300s in the route, but ensure your nginx/proxy respects it

PostgreSQL Connection String Parameters

If using PostgreSQL, ensure your DATABASE_URL includes:

pool_max=5 — Limit connections (adjust to CPU cores)
socket_timeout=45000 — Socket timeout in ms
connect_timeout=10000 — Connection establishment timeout in ms

Example PostgreSQL URL:

postgresql://user:password@vps-host:5432/dbname?schema=public&pool_max=5&socket_timeout=45000&connect_timeout=10000

Next.js Server Configuration

If running with a proxy (nginx/Apache), ensure:

# nginx example
location /api/admin/upload {
    proxy_pass http://next-server;
    proxy_connect_timeout 60s;
    proxy_send_timeout 2700s;       # 45 minutes, matches maxDuration in the route
    proxy_read_timeout 2700s;       # 45 minutes, matches maxDuration in the route
    client_max_body_size 3G;        # Must exceed the 2.5GB app-level limit
    proxy_request_buffering off;    # Stream to the app instead of spooling to nginx disk
}

Testing

After applying these changes:

Restart your Node.js server
Test with a medium-sized video (100MB+)
Check logs: pm2 logs or your logging system
Monitor database connections: SELECT count(*) FROM pg_stat_activity; in psql

Additional Tuning

If still timing out:

Check disk I/O: iostat -x 1 on VPS
Monitor database: Check if queries are slow
Increase FFprobe timeout: Edit line in app/api/admin/upload/route.ts (currently 15000ms)
Vertical scaling: Increase VPS CPU for FFprobe processing

For very large files (>500MB):

Consider streaming directly to cloud storage (S3, etc.)
Implement chunked uploads
Use a separate transcoding queue service

3.1 KiB Raw Blame History