feat(worker): parallel multipart uploads for large artifacts
Deploy / deploy (push) Successful in 3m18s
Deploy / deploy (push) Successful in 3m18s
From the studio a single connection to Hetzner nbg1 (188 ms away) is capped near 0.1 MB/s, although a browser speed test shows 50 Mbps. Measured: 1 connection 0.13 MB/s, 8 -> 0.64, 24 -> 1.69 MB/s. At the old speed a ProRes MOV could not finish inside the 30 min request timeout. - Worker 1.7.2 uploads files >= 16 MB as S3 multipart: 8 MB parts on uploadConcurrency connections (config.json, default 16), each part retried up to 4 times; aborts the upload if a part keeps failing. - artifact-presign returns uploadId + per-part presigned URLs when asked for multipart; new E20b artifact-complete stitches or aborts. - Old workers and old servers fall back to the single PUT. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -35,6 +35,12 @@ public sealed class WorkerOptions
|
||||
/// </summary>
|
||||
public List<string> PlateSearchRoots { get; set; } = new();
|
||||
/// <summary>
|
||||
/// Parallel connections per large upload. From the studio each connection
|
||||
/// to object storage is capped near 0.1 MB/s, so throughput scales with
|
||||
/// this (measured: 1 → 0.13 MB/s, 8 → 0.64, 24 → 1.69).
|
||||
/// </summary>
|
||||
public int UploadConcurrency { get; set; } = 16;
|
||||
/// <summary>
|
||||
/// Read from the assembly, not from config.json: a hand-maintained version
|
||||
/// string is exactly the thing that goes stale, and a machine reporting a
|
||||
/// version it is not running is worse than no version at all.
|
||||
|
||||
Reference in New Issue
Block a user