feat(render-pipeline): validation, QC and delivery builder
Completes RenderPipeline2 phases 3-5. Additive throughout: with validation.enabled=false the pipeline behaves exactly as before. Validation (§8) runs on the worker straight after a render, while it still holds the job and the files are local. Everything is read from the EXR header — one 64 KB read per sampled frame, no pixel decode — so the engine needs no oiiotool. The render job deliberately stays open through the stage, so a worker that dies there is reaped and retried instead of stranding the export in VALIDATING. Timecode is the primary check and is anchored on the plate, never the API: measured on the live show, Shot.timecodeStart is the editorial in-point and sits one handle-length (8f) ahead of the plate, so checking against it would pass the defective exports and fail the correct ones. Reference precedence is manifest timing plate -> plate found on disk -> API value, and the API rank is WARN-only. The panel now resolves plate provenance by walking the comp's own footage layers, so the worker never guesses which element carries the comp's timing. Verified against real studio renders via the new --validate mode: it catches both defect classes seen in shipped work — the +8 handle offset and frame-derived timecode (00:00:41:17 = frame 1001 at 24fps). QC (§10) is one service behind two front doors, API-key for the panel and session for the web. PASS means ready to upload and nothing else; FAIL sets the comp task to CHANGES. The panel gains a QC section that imports the real EXR sequence at full res rather than the H.264. Delivery (§11) rides the existing queue as a DELIVERY_BUILD job: hard links with a copy fallback, verification by re-hashing delivered bytes against the validation-time digest, manifest.json and the CSV report. fsutil hardlink list is unavailable over SMB, so link identity is never queried — reading the bytes back covers links and copies alike. Also fixes ApiClient.ProgressAsync throwing on transport failure, which would have aborted work in flight during a brief server outage; progress reporting is best-effort by contract. Removes the stale RenderWorker/ rule from the root .gitignore. The worker's source is tracked like the rest of the repo and its own .gitignore already covers bin/, obj/ and publish/; the rule silently excluded new files while the existing 20 stayed tracked. Not yet applied: migration 20260813000000 was hand-written because the dev database was unavailable — run prisma migrate deploy and verify before relying on it. None of this has run against a live server. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
+64
-3
@@ -579,6 +579,19 @@ Uploaded by the AE panel on Queue Export; snapshotted verbatim into `RenderJob.m
|
||||
"alpha": false,
|
||||
"timecodeStart": "00:59:49:12"
|
||||
},
|
||||
"source": {
|
||||
"startSource": "footage metadata",
|
||||
"handleFrames": 8,
|
||||
"timingPlate": {
|
||||
"path": "V:/_FOOTAGE/UNG/UNG_106/UNG_106_010_020_BG01_TT_v001/UNG_106_010_020_BG01_TT_v001.01001.exr",
|
||||
"frameStart": 1001,
|
||||
"frameEnd": 1120
|
||||
},
|
||||
"plates": [
|
||||
{ "element": "BG01_TT_v001", "path": "V:/_FOOTAGE/UNG/UNG_106/UNG_106_010_020_BG01_TT_v001/UNG_106_010_020_BG01_TT_v001.01001.exr", "frameStart": 1001, "frameEnd": 1120 },
|
||||
{ "element": "FG01_TT_v001", "path": "V:/_FOOTAGE/UNG/UNG_106/UNG_106_010_020_FG01_TT_v001/UNG_106_010_020_FG01_TT_v001.01001.exr", "frameStart": 1001, "frameEnd": 1120 }
|
||||
]
|
||||
},
|
||||
"preview": {
|
||||
"template": "default",
|
||||
"burnins": true
|
||||
@@ -588,6 +601,14 @@ Uploaded by the AE panel on Queue Export; snapshotted verbatim into `RenderJob.m
|
||||
|
||||
The server validates the manifest with Zod, cross-checks `shotId`/`shotCode`/frame range against the DB, and **the server (not the panel) decides the new version number** — race-free single source of truth.
|
||||
|
||||
**`source` (required from manifestVersion 2)** — the plate provenance the timecode check depends on (§8.1a). The panel resolves these from the *comp's own footage layers*, not by re-deriving paths, so the manifest records what was actually comped:
|
||||
|
||||
- `timingPlate` — the element the comp's start time was anchored to (the panel's `primaryFootage`: longest duration, first on tie). A shot's elements routinely carry unrelated camera timecodes, so "the plate" must be named explicitly rather than guessed by the worker.
|
||||
- `startSource` — verbatim from the panel's existing start-frame resolution: `"footage metadata"` (probed from the plate, trustworthy) or `"API timecode …"` (fallback, suspect). This is known at queue time, so an export built from the API fallback can be flagged **before** it burns render hours.
|
||||
- `plates[]` — every element imported into the comp, for the record and for cross-element diagnostics.
|
||||
|
||||
Manifests without `source` (manifestVersion 1) are still accepted; the timecode check degrades per §8.1a.
|
||||
|
||||
### 6.3 Endpoint summary
|
||||
|
||||
| # | Method | Path | Auth | Caller | Purpose |
|
||||
@@ -1009,11 +1030,47 @@ Runs on the worker immediately after a successful render (Export = VALIDATING).
|
||||
| `exr_type` | header: scanline vs tiled, compression | ≠ `expected.exrCompression` (WARN if compression differs but readable; FAIL if not valid EXR) |
|
||||
| `alpha_channel` | channel list contains A | presence ≠ `expected.alpha` |
|
||||
| `colourspace` | EXR metadata (`chromaticities` / ACES container flag / colorspace attr written by AE OCIO) | ≠ `expected.colorspace` (working space is linear ACES2065-1 per 18.1-Q4); `colorPipeline` is required per project (18.1-Q12), so SKIPPED only on legacy projects predating pipeline enablement |
|
||||
| `timecode_consistency` | EXR `timeCode` attr across sampled frames | non-monotonic, or first frame ≠ `expected.timecodeStart` when set |
|
||||
| `timecode_plate_match` | **primary check — see §8.1a** | render `timeCode` ≠ the timing plate's `timeCode` at the corresponding frame |
|
||||
| `timecode_consistency` | EXR `timeCode` attr across sampled frames | non-monotonic (gap, repeat, or frozen value) |
|
||||
| `timecode_fps` | EXR `framesPerSecond` rational + drop-frame flag on render vs timing plate vs manifest `fps` | any disagreement |
|
||||
| `zero_byte / corrupt` | every file size > 0; oiiotool header-read on sampled frames (first, last, every Nth) | any zero-byte or unreadable file |
|
||||
|
||||
Full-sequence checks (count, gaps, names, sizes) run on every file; header checks sample first/middle/last + every 25th frame (fast, catches systemic issues; per-frame full decode is not worth minutes per shot — tunable via `SystemConfig: validation.sampleEvery`).
|
||||
|
||||
### 8.1a Timecode: the primary check (studio decision)
|
||||
|
||||
Timecode is the check this engine exists for. It is validated **against the source plate**, never against the API, because the API and the plate do not agree by design.
|
||||
|
||||
**Measured on the live show (2026-08-13, 85 rendered sequences + 106 delivered sequences on `V:/`):**
|
||||
|
||||
- AE-written EXRs *do* carry `timeCode` (plus a `timeCodeString` mirror) and `framesPerSecond` — the §17.3 "metadata may be absent" risk does not apply to the studio's current EXR output module. A render whose TC is absent means the output module changed, and is a FAIL.
|
||||
- `Shot.timecodeStart` from the API is the **editorial in-point**; the plate on disk starts `handleFrames` earlier. Measured on three shots, the API value was consistently the plate's TC **+8 frames** — the exact handle count. Validating a render against the API would therefore **pass the defective exports and fail the correct ones**: on `UNG_102_019_010` the correctly-built render (`18:58:57:12`, matching its BG01 plate) differs from the API value (`18:58:57:20`), while on `UNG_102_040_010` the *defective* render matches the API exactly and is 8 frames off its plate.
|
||||
- 95 of 106 delivered sequences match their plate's timecode exactly. The 11 that do not fall into three classes this check catches: **+8 handle offset** (comp anchored to the editorial in-point instead of the plate head), **frame-derived TC** (`00:00:41:17` = frame 1001 at 24 fps, i.e. a comp with no source timecode at all — also seen as `00:00:00:00`), and **one timecode shared across four different shots** in a single delivery package.
|
||||
|
||||
**Reference precedence** — the check records which source it used and never silently substitutes a weaker one:
|
||||
|
||||
| Rank | Source | Verdict semantics |
|
||||
|---|---|---|
|
||||
| 1 | `source.timingPlate` from the manifest — the element the comp was actually anchored to | PASS / FAIL, authoritative |
|
||||
| 2 | Plate re-derived on the worker from the footage root + shot code (manifestVersion 1, or a timing plate that no longer resolves). If elements disagree, the one matching the render's frame count wins; still ambiguous → WARN, never PASS | PASS / FAIL, `reference: "derived"` |
|
||||
| 3 | API `Shot.timecodeStart` **minus `handleFrames`** | **WARN only, never PASS** — `reference: "api"`, message states the plate could not be read |
|
||||
|
||||
Rank 3 is a diagnostic, not a verification: it depends on the show's handle count being correct for that specific plate, which is exactly the assumption the defects above violate.
|
||||
|
||||
**Comparison, given the plate reference:**
|
||||
|
||||
```
|
||||
expectedTc = plateTc + (render.frameStart - plate.frameStart) // frames, at manifest fps
|
||||
```
|
||||
|
||||
The offset term matters — a shot may render a sub-range of a long plate (observed: 37 rendered frames from a 709-frame plate). Sampled frames must then increment by exactly one frame each; a constant value across the sequence is the frame-derived-TC defect, not a rounding artifact. `timeCodeString` is cross-checked against the packed `timeCode` as a writer sanity check.
|
||||
|
||||
**Queue-time pre-flight.** The panel already knows whether it read the start time from the plate's metadata or fell back to the API (`source.startSource`). When the manifest says the API fallback was used, the Export is flagged at queue time — the render is very likely to fail this check, and there is no point spending machine hours to find out.
|
||||
|
||||
**Delivery-package cross-check (Phase 5).** Within one package, two shots sharing a first-frame timecode is a FAIL on the package: it is cheap, and it is precisely the defect that reached a built delivery package above.
|
||||
|
||||
**Tooling note.** `timeCode`, `timeCodeString` and `framesPerSecond` live in the EXR header, so this check needs one 64 KB read per sampled frame and no pixel decode — the header parse is ~40 lines (attribute list: `name\0 type\0 int32 size data`; `timeCode` is two `uint32`, the first packed SMPTE 12M BCD, which reads back as its own hex digits: `0x18585712` → `18:58:57:12`). It does **not** depend on oiiotool, so the timecode check can ship ahead of the OIIO-based resolution/bit-depth/channel checks.
|
||||
|
||||
### 8.2 Semantics
|
||||
|
||||
- Any FAIL → whole run FAIL → Export = VALIDATION_FAILED, pipeline stops, exact reasons in `ValidationResult` rows (E12 example above shows a missing-frame report).
|
||||
@@ -1126,7 +1183,11 @@ Tokens: `{showId} {episode} {yymmdd} {yyyymmdd} {packageName} {packageDir} {shot
|
||||
|
||||
### 11.3 Hard links vs copying
|
||||
|
||||
**Recommendation: hard link when `deliveryRoot` and render root share an NTFS volume; transparent fallback to copy otherwise (cross-volume links are impossible).** Hard links make an 18-shot EXR package near-instant and ~zero extra disk; the risk (mutating a linked file changes both) is acceptable because rendered EXR versions are immutable by pipeline design — a version is never re-rendered in place (new version → new directory). Post-link verification: link target identity check; post-copy verification: size + xxHash against validation-time values. `DeliveryItem.status` → VERIFIED per item. Config override `"copyMode": "always-copy"` for productions that mandate physical separation.
|
||||
**Recommendation: hard link when `deliveryRoot` and render root share an NTFS volume; transparent fallback to copy otherwise (cross-volume links are impossible).** Hard links make an 18-shot EXR package near-instant and ~zero extra disk; the risk (mutating a linked file changes both) is acceptable because rendered EXR versions are immutable by pipeline design — a version is never re-rendered in place (new version → new directory).
|
||||
|
||||
**Verified on the studio volume (2026-08-13):** `V:` is `\\TRUENAS\vfx` over SMB, and hard links work on it — `mklink /H` succeeds and a write through one name is visible through the other. Renders (`V:/_EXPORTS/UNG/{ep}/{shot}/{ver}`) and the delivery root (`V:/_EXPORTS/UNG/DELIVERY`) are on the same share, so the fast path applies to real deliveries today.
|
||||
|
||||
**Implementation trap:** `fsutil hardlink list` fails over SMB (`Error 50: The request is not supported`), so the builder **cannot enumerate links to verify them** on a network volume. Verify instead by hashing the delivery-side file and comparing against the validation-time xxHash — that reads the same bytes whether the entry is a link or a copy, so one verification path covers both modes. Post-copy verification is the same check plus size. `DeliveryItem.status` → VERIFIED per item. Config override `"copyMode": "always-copy"` for productions that mandate physical separation.
|
||||
|
||||
### 11.4 Manifest and delivery report
|
||||
|
||||
@@ -1278,7 +1339,7 @@ Five independently deployable phases. Each phase leaves the previous workflow fu
|
||||
| API | E12; transitions T3→T7/T8 activated (Phase 2 interim: complete-render goes straight to a provisional READY_FOR_QC; Phase 3 inserts VALIDATING between) |
|
||||
| UI | Validation table on Export detail; Validation Results browser; failure badge in queue |
|
||||
| Worker | Validation engine (oiiotool integration, all Section 8.1 checks, checksums); oiiotool bundled with installer |
|
||||
| Risks | Colourspace/timecode metadata may be absent in AE-written EXRs → those checks SKIPPED until per-project expectations configured (18.2-Q1); false FAILs block artists → per-check severity override in SystemConfig (`validation.overrides: {"exr_type": "warn"}`) |
|
||||
| Risks | ~~Timecode metadata may be absent in AE-written EXRs~~ — disproved on the live show (§8.1a): AE writes `timeCode`, so absence is now itself a FAIL. Colourspace metadata may still be absent → that check SKIPPED until per-project expectations configured (18.2-Q1); false FAILs block artists → per-check severity override in SystemConfig (`validation.overrides: {"exr_type": "warn"}`). **The timecode check is the phase's priority and ships first** — it needs no oiiotool, and 11 of 106 delivered sequences on the current show would have been caught by it |
|
||||
| Testing | Golden EXR fixture sets: complete, gapped, duplicate, wrong-res, wrong-depth, no-alpha, corrupt, zero-byte; checksum determinism; large-sequence perf (500 frames < 60 s) |
|
||||
| Migration | Additive; existing in-flight exports unaffected (transition map keys off manifest version) |
|
||||
|
||||
|
||||
Reference in New Issue
Block a user