feat(worker): parallel multipart uploads for large artifacts
Deploy / deploy (push) Successful in 3m18s
Deploy / deploy (push) Successful in 3m18s
From the studio a single connection to Hetzner nbg1 (188 ms away) is capped near 0.1 MB/s, although a browser speed test shows 50 Mbps. Measured: 1 connection 0.13 MB/s, 8 -> 0.64, 24 -> 1.69 MB/s. At the old speed a ProRes MOV could not finish inside the 30 min request timeout. - Worker 1.7.2 uploads files >= 16 MB as S3 multipart: 8 MB parts on uploadConcurrency connections (config.json, default 16), each part retried up to 4 times; aborts the upload if a part keeps failing. - artifact-presign returns uploadId + per-part presigned URLs when asked for multipart; new E20b artifact-complete stitches or aborts. - Old workers and old servers fall back to the single PUT. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
+1
-1
@@ -1194,7 +1194,7 @@ Renders run past the cut at each end (8 + 8 by house convention). Handles ship w
|
||||
The Standard review MP4 is AE's H.264 (UHD, ~35 Mbps). ProRes itself can't be the player's source — only Safari on a Mac decodes it in a web page, and UHD ProRes LT is ~280 Mbps — so the worker (≥ 1.7.0, when `preview.highQuality` is on) adds two things after the AE render, both **best-effort** (a failure costs the extras, never the preview):
|
||||
|
||||
- **High quality MP4:** ffmpeg re-encodes the delivery MOV as full-res H.264 High, CRF 14 capped at 100 Mbps, yuv420p, faststart → `Version.hqFileUrl`. The player shows an **HQ** switch when present; switching keeps frame, play state, speed and zoom, is remembered per browser (`review:quality`), and falls back to Standard if the file won't load. Same comp as the Standard MP4, so slate and handles line up frame for frame.
|
||||
- **ProRes download:** the MOV itself is uploaded (presign kind `highres`) → `Version.highResKey/highResFilename/highResSize`. It is the version's **High Res** download in the client portal (named with the client code, `clientHighResFilename`) and a **ProRes** button on the staff review page. The shot's manually uploaded High Res remains the fallback for versions without one. Since worker 1.7.1 the MOV uploads **in the background after finalize** (`HighResUploader`, queue in `%ProgramData%\VFXReviewWorker\highres-queue`, resumes after a restart, 5 attempts) and attaches via `POST /api/ext/render/jobs/{id}/attach-highres` (E13b); a MOV from a preview that a newer preview has superseded is discarded. While a preview job uploads, the worker replaces the job's log tail with live upload status.
|
||||
- **ProRes download:** the MOV itself is uploaded (presign kind `highres`) → `Version.highResKey/highResFilename/highResSize`. It is the version's **High Res** download in the client portal (named with the client code, `clientHighResFilename`) and a **ProRes** button on the staff review page. The shot's manually uploaded High Res remains the fallback for versions without one. Since worker 1.7.1 the MOV uploads **in the background after finalize** (`HighResUploader`, queue in `%ProgramData%\VFXReviewWorker\highres-queue`, resumes after a restart, 5 attempts) and attaches via `POST /api/ext/render/jobs/{id}/attach-highres` (E13b); a MOV from a preview that a newer preview has superseded is discarded. While a preview job uploads, the worker replaces the job's log tail with live upload status. Since 1.7.2, artifacts of 16 MB or more upload as S3 multipart: `artifact-presign` with `multipart: true, size` returns `uploadId`, `partSize` (8 MB minimum) and one presigned URL per part; the worker PUTs parts on `uploadConcurrency` connections (default 16), retries each part up to 4 times, then calls `artifact-complete` (E20b) to stitch them, or with `abort: true` to discard them. A single connection from the studio to nbg1 is capped near 0.1 MB/s.
|
||||
|
||||
A lease keep-alive runs during the encode and uploads (they can outlast the 300 s lease). The export timeline records what was attached.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user