diff --git a/DELIVERY_WORKFLOW.md b/DELIVERY_WORKFLOW.md new file mode 100644 index 0000000..451f09c --- /dev/null +++ b/DELIVERY_WORKFLOW.md @@ -0,0 +1,219 @@ +# Delivery Packages — Workflow Guide + +How a finished shot becomes a dated folder ready to upload to the client. + +This is the operational guide. The design rationale and normative rules live in +[RenderPipeline2.md](RenderPipeline2.md) §11. + +--- + +## The short version + +1. A shot renders, validates, gets a preview, and **passes QC** → it becomes + eligible for delivery. +2. On **Pipeline → Deliveries**, pick the project, see *"Episode 101 — Ready: + 18 shots"*, click **Build delivery**. +3. A render worker builds the folder on the delivery volume, verifies every + file, and writes a manifest and CSV report. +4. A human uploads that folder to the client, then clicks **Mark delivered**. + +Nothing is ever sent to the client automatically. Step 4 is deliberately manual. + +--- + +## What makes a shot eligible + +An Export reaches `READY_FOR_DELIVERY` only by passing QC — from the AE panel +or the web. That is the single gate; there is no way to package an export that +has not been QC'd. + +Two rules worth knowing: + +- **Newest version per shot only.** If a shot has two exports sitting ready, the + older one is history and is not offered. Building refuses outright if you + somehow select two versions of the same shot. +- **Version locking.** A package references the exact Export rows it was built + from and copies the shot code and version string at creation time. Re-render a + shot afterwards and the package is untouched — delivery history is immutable + by construction. + +--- + +## Building a package + +**Pipeline → Deliveries**, choose a project. Each episode with ready shots shows +its count and the shots themselves, with frame counts and sizes. + +**Build delivery** creates the package (`QUEUED`) and a `DELIVERY_BUILD` job on +the same queue the renders use. Requires **admin, producer or supervisor**. + +Delivery jobs run at priority 10, which is inside the urgent threshold — so they +claim immediately even during work hours, when a render would wait for the +evening window. Packaging is file work with no After Effects involvement, so it +does not disturb an artist at the machine. + +Watch the package status on the same page: + +| Status | Meaning | +|---|---| +| `QUEUED` | Waiting for a worker to pick it up | +| `BUILDING` | A worker is creating folders and linking files | +| `READY` | Built and verified — upload it | +| `DELIVERED` | You confirmed the client has it | +| `FAILED` | Something did not verify — see the error and the per-shot list | +| `CANCELLED` | Abandoned before delivery | + +--- + +## What lands on disk + +Default layout (the studio's confirmed structure): + +``` +{deliveryRoot}/{episode}/{YYMMDD}_Delivery/ +├── {SHOT_CODE}/ +│ ├── {shot}_cmp_TT_{ver}.[#####].exr ← the EXR sequence +│ └── {shot}_cmp_TT_{ver}.mov ← slate/burn-in delivery MOV +├── manifest.json ← every file, size, checksum, source export +└── {YYMMDD}_Delivery_report.csv ← one row per shot (client-facing) +``` + +EXRs and the MOV sit side by side in each shot folder — no `exr/` and `mov/` +subfolders, matching current practice. + +The CSV carries: shot, version, frame range, frame count, file count, bytes, +checksum, MOV filename and status. Confirm these columns against the +production's delivery spec before the first real delivery. + +--- + +## Hard links, and why the package appears instantly + +Files are **hard-linked**, not copied, when the source and destination are on the +same volume. A hard link is a second name for the same data — not a shortcut and +not a copy — so an 18-shot EXR package builds in seconds and consumes +essentially no extra disk. + +Verified on the studio SAN (`\\TRUENAS\vfx` over SMB): hard links work there, and +renders and deliveries live on the same share. + +**Is that safe?** Deleting the delivery folder does not delete the render — the +data survives until the last name referring to it is gone. And nothing in the +pipeline ever rewrites an existing EXR: a new version is always a new directory. + +Two escape hatches: + +- A delivery root on a **different volume** falls back to copying automatically. +- `"copyMode": "always-copy"` in the project's delivery config forces real copies + for a production that mandates physical separation. + +The `linkMode` column on each item records what actually happened, so a package +never claims to have linked when it copied. + +--- + +## Verification + +Every file is read back after packaging and hashed, and the sequence digest is +compared against the one recorded at validation time. `fsutil hardlink list` does +not work over SMB, so link identity is never queried — reading the bytes covers +links and copies through one path. + +An item fails, and the whole package fails with it, when: + +| Failure | Meaning | +|---|---| +| Checksum mismatch | The delivered bytes differ from what passed QC | +| Frame count differs from the export record | The render folder changed after QC — someone re-rendered in place | +| Source folder missing | The render is not where the export says it is | +| No matching files | The folder exists but holds nothing matching the output pattern | + +**A package only reaches `READY` when every shot verified.** One bad shot means +`FAILED`, and no export is marked as packaged — there is no partial delivery. + +A missing delivery MOV is the one non-fatal case: the EXRs are the deliverable, +so it is logged and the package continues without it. + +> **Without validation there is nothing to verify against.** An export whose +> checksum was never recorded (validation disabled, or rendered before it +> shipped) still packages, but the digest is computed with nothing to compare it +> to. The ready list flags these as *"N without a validation checksum"*. + +--- + +## Marking delivered + +Once you have uploaded the folder, **Mark delivered** moves every packaged +export to `DELIVERED`. Requires admin, producer or supervisor. + +This is the end of the pipeline, and it is a human statement of fact — the app +has no way to know the client received anything. + +--- + +## When something goes wrong + +**A package failed.** Expand it: every shot shows its own status and error. Fix +the cause (usually a re-render after QC), then build a new package. Packages are +never deleted; a mistaken one is cancelled. + +**The worker died mid-build.** The package is marked `FAILED` with *"check the +package folder before rebuilding"*. It is deliberately not retried +automatically — a half-written folder on the delivery volume is something a +human should look at first. + +**Cancelling.** Only `DRAFT`, `QUEUED` and `FAILED` packages can be cancelled. A +`READY` or `DELIVERED` package stays in the history permanently; supersede it by +building a new one. + +--- + +## Configuration + +Per project, in `Project.deliveryConfig`; studio-wide defaults in the +`delivery.defaultConfig` SystemConfig key; built-in defaults otherwise. Merged +key by key, so a project can override just `deliveryRoot`. + +```json +{ + "packageNameTemplate": "{yymmdd}_Delivery", + "layout": { + "packageDir": "{deliveryRoot}/{episode}/{packageName}", + "shotDir": "{packageDir}/{shotCode}", + "reportFile": "{packageDir}/{packageName}_report.csv" + }, + "deliveryRoot": "//SAN/DELIVERY", + "includeMovs": true, + "checksumAlgo": "xxh64", + "reportFormats": ["csv", "json"], + "copyMode": "auto" +} +``` + +Tokens: `{showId} {episode} {yymmdd} {yyyymmdd} {packageName} {packageDir} +{deliveryRoot} {shotCode} {versionString} {packageNumber}`. + +The resolved config is **snapshotted onto the package** at build time, so a +historic package always shows the rules it was built with, and a production with +a different spec is a config change rather than a code change. + +`deliveryRoot` must be reachable *from the worker machine* — check with +`VFXReviewWorker.exe --check` if a build fails on paths. + +--- + +## API + +| # | Method | Path | Purpose | +|---|--------|------|---------| +| E16 | `POST` | `/api/deliveries` | Create a package and queue the build | +| E17 | `GET` | `/api/deliveries` | List packages / history | +| E18 | `GET` | `/api/deliveries/ready` | Ready exports grouped by episode | +| E19 | `POST` | `/api/deliveries/{id}/delivered` | Mark delivered | +| — | `POST` | `/api/deliveries/{id}/cancel` | Cancel a package | +| — | `POST` | `/api/ext/render/jobs/{jobId}/delivery` | Worker reports a build (API key) | + +`POST /api/deliveries` takes `{ projectId, episode?, exportIds?, options? }`. +Omit `exportIds` to take every ready export in the episode; `options` accepts +`date`, `name`, `includeMovs` and `copyMode` to override the config for one +build. diff --git a/README.md b/README.md index 1fe930a..593350e 100644 --- a/README.md +++ b/README.md @@ -166,6 +166,19 @@ prisma/ — schema.prisma, seed.ts --- +## Documentation + +| Doc | Covers | +|---|---| +| [RenderPipeline2.md](RenderPipeline2.md) | Render pipeline specification — normative design, state machine, API contracts | +| [DELIVERY_WORKFLOW.md](DELIVERY_WORKFLOW.md) | Building and shipping delivery packages (how-to) | +| [VFXReviewConnector.md](VFXReviewConnector.md) | After Effects panel — setup, Shot Builder, QC, diagnostics | +| [RenderWorker/README.md](RenderWorker/README.md) | Render worker service — packaging, install, validation, delivery builds | +| [EXT_API_REFERENCE.md](EXT_API_REFERENCE.md) | External API (`/api/ext/*`) used by the panel and workers | +| [Shot task workflow.md](Shot%20task%20workflow.md) | Shot and task lifecycle in the web app | + +--- + ## License MIT diff --git a/RenderWorker/README.md b/RenderWorker/README.md index cb06021..013c09d 100644 --- a/RenderWorker/README.md +++ b/RenderWorker/README.md @@ -89,6 +89,24 @@ Server-side switches (SystemConfig, no worker redeploy): `validation.enabled`, `validation.sampleEvery`, `validation.handleFrames`, and `validation.overrides` for per-check severity, e.g. `{"exr_type":"warn"}`. +## Delivery builds + +The same service also claims `DELIVERY_BUILD` jobs: creating a dated package +folder, hard-linking (or copying) each shot's EXRs and delivery MOV into it, +verifying every file against the validation-time checksum, and writing +`manifest.json` plus the CSV report. + +No After Effects involved, so these are claimed **even while an artist has AE +open** — and at priority 10, inside the urgent threshold, so they run during the +day rather than waiting for the render window. Someone is usually waiting on the +folder. + +`deliveryRoot` has to be reachable from this machine under the service account: +`--check` verifies `pathMappings` targets, which is where a build most often +falls over. + +Full workflow: [DELIVERY_WORKFLOW.md](../DELIVERY_WORKFLOW.md). + ### Checking a render by hand ```powershell