feat(worker): parallel multipart uploads for large artifacts
Deploy / deploy (push) Successful in 3m18s

From the studio a single connection to Hetzner nbg1 (188 ms away) is
capped near 0.1 MB/s, although a browser speed test shows 50 Mbps.
Measured: 1 connection 0.13 MB/s, 8 -> 0.64, 24 -> 1.69 MB/s. At the old
speed a ProRes MOV could not finish inside the 30 min request timeout.

- Worker 1.7.2 uploads files >= 16 MB as S3 multipart: 8 MB parts on
  uploadConcurrency connections (config.json, default 16), each part
  retried up to 4 times; aborts the upload if a part keeps failing.
- artifact-presign returns uploadId + per-part presigned URLs when asked
  for multipart; new E20b artifact-complete stitches or aborts.
- Old workers and old servers fall back to the single PUT.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
twotalesanimation
2026-10-01 18:48:28 +02:00
parent 477aef7b27
commit 91e69f0bcb
11 changed files with 530 additions and 21 deletions
+3
View File
@@ -208,6 +208,9 @@ read back from the assembly, so a machine can never report a version it is not r
`pathMappings` translate the manifest's canonical UNC paths to this machine's drive
mappings. `plateSearchRoots` is optional: it is only consulted when a manifest carries no
timing plate (older panel versions), to find the shot's plates for the timecode check.
`uploadConcurrency` (default 16) is how many parallel connections a file of 16 MB or more
uploads over. Each connection to object storage is capped near 0.1 MB/s from the studio, so
throughput scales with it (measured 2026-10-01: 1 → 0.13 MB/s, 8 → 0.64, 24 → 1.69 MB/s).
Everything tunable (poll interval, lease, stall timeout, max attempts, validation) lives in
the server's SystemConfig and arrives at registration — no per-machine tuning files.