feat(worker): parallel multipart uploads for large artifacts
Deploy / deploy (push) Successful in 3m18s
Deploy / deploy (push) Successful in 3m18s
From the studio a single connection to Hetzner nbg1 (188 ms away) is capped near 0.1 MB/s, although a browser speed test shows 50 Mbps. Measured: 1 connection 0.13 MB/s, 8 -> 0.64, 24 -> 1.69 MB/s. At the old speed a ProRes MOV could not finish inside the 30 min request timeout. - Worker 1.7.2 uploads files >= 16 MB as S3 multipart: 8 MB parts on uploadConcurrency connections (config.json, default 16), each part retried up to 4 times; aborts the upload if a part keeps failing. - artifact-presign returns uploadId + per-part presigned URLs when asked for multipart; new E20b artifact-complete stitches or aborts. - Old workers and old servers fall back to the single PUT. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,18 @@
|
||||
/**
|
||||
* Worker artifact kinds (E20) and the object-storage folder each lands in:
|
||||
* preview → videos/, thumbnail → image/, metadata/log → renders/, highres (the
|
||||
* delivery MOV) → highres/ beside the manually uploaded High Res files.
|
||||
*/
|
||||
export const ARTIFACT_KIND_FOLDERS: Record<string, string> = {
|
||||
preview: "videos",
|
||||
thumbnail: "image",
|
||||
metadata: "renders",
|
||||
log: "renders",
|
||||
highres: "highres",
|
||||
};
|
||||
|
||||
/** True for keys artifact-presign could have issued — guards E20b. */
|
||||
export function isArtifactKey(key: string): boolean {
|
||||
const folder = key.split("/")[0];
|
||||
return key.includes("/") && !key.includes("..") && Object.values(ARTIFACT_KIND_FOLDERS).includes(folder);
|
||||
}
|
||||
Reference in New Issue
Block a user