feat(pipeline): render queue, worker service and automated preview generation
One "Queue Export" click now renders the EXR sequence, then rebuilds the shot headlessly with the studio slate/overlay template to produce the delivery MOV and review MP4. Implements RenderPipeline2 phases 1-2 plus the preview stage. Server: - New models Export, RenderJob, ExportEvent, Machine, WorkerHeartbeat, plus Project.deliveryConfig and per-submission slate fields (Export.vfxScope, Export.submissionNote, inherited from the shot's previous export). Both migrations are purely additive; no existing column is touched. - lib/render-pipeline: server-enforced state machine, transactional version increment with supersede, atomic FOR UPDATE SKIP LOCKED claim gated by machine availability windows, and a lease reaper run from instrumentation.ts. - /api/ext/* endpoints for the panel and workers; session-auth mirrors under /api/render and /api/machines for the web UI. - Pipeline pages: render queue, export detail, machine monitoring, plus an Exports tab on shot detail. RenderWorker (.NET 8 Windows service, new): - Registration, heartbeat as cancel channel, claim loop, aerender runner with progress parsing and stall watchdog, crash recovery and disk-spooled reporting that survives server downtime. - Preview stage: headless AE assembles the preview comp into a throwaway AEP with both output modules queued, then a single aerender pass renders them. Preview jobs are not claimed while an interactive AE session is open, so an artist's project is never taken over. AE panel: Queue Export with live status polling, urgent flag, retry, and the VFX Scope / Submission Note fields. Every existing panel action is unchanged. Preview chaining ships disabled behind SystemConfig preview.enabled. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
twotalesanimation
@@ -0,0 +1,137 @@
|
||||
# VFXReview RenderWorker
|
||||
|
||||
Windows service that drives `aerender.exe` on the artist workstations (RenderPipeline2 §7).
|
||||
It talks **only HTTP** to the VFXReview server (`/api/ext/*`, API-key auth) — it never
|
||||
gets a database connection. All scheduling policy (render windows, Render Now,
|
||||
urgent priority) is enforced server-side in the claim endpoint; the worker polls dumbly.
|
||||
|
||||
## One click → EXR + MOV + MP4
|
||||
|
||||
A single **Queue Export** click produces all three deliverables. The server
|
||||
chains two jobs:
|
||||
|
||||
1. **AE_RENDER** — `aerender` writes the clean EXR sequence (no overlay, no LUT).
|
||||
2. **PREVIEW_ONLY** — created automatically when the render completes. The
|
||||
worker runs `AfterFX.com -noui -r scripts/vfxr_build_preview.jsx`, which opens
|
||||
the studio slate/overlay **template** AEP, imports the rendered EXRs, rebuilds
|
||||
the shot around them (OCIO → `_SHOW LUT` → `UNG_VFX_OVERLAY`), duplicates
|
||||
`UNG_EXPORT_TEMPLATE` into a preview comp with the slate filled in, queues the
|
||||
MOV (`4444 Tri`) and MP4 (`REVIEW_PREVIEW`) output modules, and **saves a
|
||||
throwaway AEP**. The worker then runs `aerender -project <that aep>` with no
|
||||
`-comp`, rendering both outputs in one launch.
|
||||
|
||||
The MOV and MP4 land beside the EXRs. The MP4 is uploaded and registered as an
|
||||
ordinary `Version` — internal-only, never client-visible, and it changes no task
|
||||
or shot status (§10.0). Preview settings live in `SystemConfig` under
|
||||
`preview.*`, so template names and paths are changed without redeploying.
|
||||
|
||||
If the preview stage fails, only it is retried — the validated EXRs are never
|
||||
re-rendered. The temp AEP is kept on failure so you can open it and see exactly
|
||||
what the farm built.
|
||||
|
||||
### Proving headless AE first
|
||||
|
||||
Headless AE is the one real unknown, so prove it before relying on it
|
||||
(spec 18.2-C4). **Close After Effects**, then:
|
||||
|
||||
```powershell
|
||||
.\scripts\test-preview-build.ps1 -ExrDir "V:\_EXPORTS\...\v002" -ShotCode "UNG_111_001_030" -Version "v002"
|
||||
```
|
||||
|
||||
It runs the exact build the worker runs and prints the result plus any warnings
|
||||
(missing LUT comp, missing slate layer, …), then gives you the `aerender` command
|
||||
to render what it built. If it reports no result file, AE cannot script headlessly
|
||||
under that account — fall back to having the panel pre-build the preview comp in
|
||||
the artist's AEP (then it is pure `aerender`), or the ffmpeg engine.
|
||||
|
||||
## What it does (Phase 2 scope)
|
||||
|
||||
- Registers on startup (E6) and receives server-supplied tuning (poll/heartbeat/lease/stall).
|
||||
- Heartbeats every 30 s (E7) — the heartbeat response is also the cancel channel.
|
||||
- Claims one `AE_RENDER` job at a time (E8), runs `aerender.exe` with the manifest's
|
||||
comp/frame-range/templates, parses `PROGRESS:` lines, reports progress + ETA (E9,
|
||||
renews the lease).
|
||||
- Stall watchdog: no progress for `stallTimeoutSeconds` (default 600) → kill process tree, retryable fail.
|
||||
- Deterministic errors (missing footage / missing comp / unopenable project) → non-retryable fail (E10);
|
||||
transient errors auto-requeue server-side up to `maxAttempts`.
|
||||
- On success: uploads the full aerender log via presign (E20), reports complete (E11).
|
||||
- Crash recovery: `current-job.json` written on claim; on restart the worker asks the server
|
||||
what became of the job and reports a retryable fail if it was still ours. Partial renders
|
||||
are never resumed — the next attempt clears its own output files and re-renders.
|
||||
- Durable reporting: complete/fail reports spool to disk and replay in order with backoff —
|
||||
rendering continues while the server is down.
|
||||
|
||||
Preview generation (Phase 4) and validation (Phase 3) plug into this same service later.
|
||||
|
||||
## Build
|
||||
|
||||
Requires the .NET 8+ SDK.
|
||||
|
||||
```bash
|
||||
cd RenderWorker/VFXReviewWorker
|
||||
dotnet publish -c Release -r win-x64 --self-contained -p:PublishSingleFile=true -o publish
|
||||
```
|
||||
|
||||
Produces a single `publish/VFXReviewWorker.exe` — no runtime install needed on render nodes.
|
||||
|
||||
## Configure
|
||||
|
||||
Create `C:\ProgramData\VFXReviewWorker\config.json` (§7.8):
|
||||
|
||||
```json
|
||||
{
|
||||
"serverUrl": "https://review.twotalesvfx.com",
|
||||
"apiKey": "<API_SECRET_KEY>",
|
||||
"machineName": "RENDER-01",
|
||||
"aerenderPath": "C:\\Program Files\\Adobe\\Adobe After Effects 2026\\Support Files\\aerender.exe",
|
||||
"aeVersion": "24.3",
|
||||
"ffmpegPath": "C:\\pipeline\\bin\\ffmpeg.exe",
|
||||
"pathMappings": [
|
||||
{ "from": "//SAN/", "to": "S:/" }
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
`pathMappings` translate the manifest's canonical UNC paths to this machine's drive
|
||||
mappings. Everything tunable (poll interval, lease, stall timeout, max attempts) lives in
|
||||
the server's SystemConfig and arrives at registration — no per-machine tuning files.
|
||||
|
||||
## Run interactively (first-time smoke test)
|
||||
|
||||
```bash
|
||||
VFXReviewWorker.exe
|
||||
```
|
||||
|
||||
Logs go to the console-less service log at `%ProgramData%\VFXReviewWorker\logs\worker_YYYYMMDD.log`
|
||||
(14-day rolling). Confirm the machine appears on the web **Pipeline → Machines** page, then stop it.
|
||||
|
||||
## Install as a Windows service
|
||||
|
||||
Run as a studio account with SAN access (works whether or not an artist is logged in):
|
||||
|
||||
```powershell
|
||||
sc.exe create VFXReviewRenderWorker binPath= "C:\pipeline\VFXReviewWorker\VFXReviewWorker.exe" start= auto obj= "STUDIO\svc-render" password= "<password>"
|
||||
sc.exe description VFXReviewRenderWorker "VFXReview render pipeline worker (aerender)"
|
||||
sc.exe start VFXReviewRenderWorker
|
||||
```
|
||||
|
||||
Uninstall: `sc.exe stop VFXReviewRenderWorker && sc.exe delete VFXReviewRenderWorker`.
|
||||
|
||||
## Rollout notes (spec §17.2)
|
||||
|
||||
- Install on **one** workstation first; add the second after a clean week.
|
||||
- Set `render.maxAttempts = 1` in SystemConfig for the first week (observe before auto-retrying).
|
||||
- Enter each machine's render windows on the Machines page (`availability`), defaults:
|
||||
weekdays 19:00–08:00 + full weekends; Render Now override = 4 h.
|
||||
- The dashboard warns when the two workstations report different AE versions.
|
||||
|
||||
## Tests
|
||||
|
||||
```bash
|
||||
cd RenderWorker
|
||||
dotnet test
|
||||
```
|
||||
|
||||
Covers the progress parser against recorded aerender transcript lines (success, error,
|
||||
non-retryable), ETA math, path mapping, log-tail ring buffer, and the output-dir
|
||||
clearing guard (only files matching the job's own pattern prefix are ever deleted).
|
||||
Reference in New Issue
Block a user