vfxreview/RenderWorker

VFXReview RenderWorker

Windows service that drives aerender.exe on the artist workstations (RenderPipeline2 §7). It talks only HTTP to the VFXReview server (/api/ext/*, API-key auth) — it never gets a database connection. All scheduling policy (render windows, Render Now, urgent priority) is enforced server-side in the claim endpoint; the worker polls dumbly.

One click → EXR + MOV + MP4

A single Queue Export click produces all three deliverables. The server chains two jobs:

1. AE_RENDER — aerender writes the clean EXR sequence (no overlay, no LUT).
2. PREVIEW_ONLY — created automatically when the render completes. The worker runs AfterFX.com -noui -r scripts/vfxr_build_preview.jsx, which opens the studio slate/overlay template AEP, imports the rendered EXRs, rebuilds the shot around them (OCIO → _SHOW LUT → UNG_VFX_OVERLAY), duplicates UNG_EXPORT_TEMPLATE into a preview comp with the slate filled in, queues the MOV (4444 Tri) and MP4 (REVIEW_PREVIEW) output modules, and saves a throwaway AEP. The worker then runs aerender -project <that aep> with no -comp, rendering both outputs in one launch.

The MOV and MP4 land beside the EXRs. The MP4 is uploaded and registered as an ordinary Version — internal-only, never client-visible, and it changes no task or shot status (§10.0). Preview settings live in SystemConfig under preview.*, so template names and paths are changed without redeploying.

If the preview stage fails, only it is retried — the validated EXRs are never re-rendered. The temp AEP is kept on failure so you can open it and see exactly what the farm built.

Proving headless AE first

Headless AE is the one real unknown, so prove it before relying on it (spec 18.2-C4). Close After Effects, then:

.\scripts\test-preview-build.ps1 -ExrDir "V:\_EXPORTS\...\v002" -ShotCode "UNG_111_001_030" -Version "v002"

It runs the exact build the worker runs and prints the result plus any warnings (missing LUT comp, missing slate layer, …), then gives you the aerender command to render what it built. If it reports no result file, AE cannot script headlessly under that account — fall back to having the panel pre-build the preview comp in the artist's AEP (then it is pure aerender), or the ffmpeg engine.

What it does (Phase 2 scope)

• Registers on startup (E6) and receives server-supplied tuning (poll/heartbeat/lease/stall).
• Heartbeats every 30 s (E7) — the heartbeat response is also the cancel channel.
• Claims one AE_RENDER job at a time (E8), runs aerender.exe with the manifest's comp/frame-range/templates, parses PROGRESS: lines, reports progress + ETA (E9, renews the lease).
• Stall watchdog: no progress for stallTimeoutSeconds (default 600) → kill process tree, retryable fail.
• Deterministic errors (missing footage / missing comp / unopenable project) → non-retryable fail (E10); transient errors auto-requeue server-side up to maxAttempts.
• On success: uploads the full aerender log via presign (E20), reports complete (E11).
• Crash recovery: current-job.json written on claim; on restart the worker asks the server what became of the job and reports a retryable fail if it was still ours. Partial renders are never resumed — the next attempt clears its own output files and re-renders.
• Durable reporting: complete/fail reports spool to disk and replay in order with backoff — rendering continues while the server is down.
• Validation (Phase 3): after a successful render the worker checks the sequence it just produced and reports every result in one call (E12). See below.

Validation stage

Runs on the worker straight after the render, while it still holds the job and the files are local. The Export sits in VALIDATING until the report lands; the server decides what the results mean (any FAIL → VALIDATION_FAILED, otherwise the preview stage is queued).

Checks: file_count, missing_frames, duplicate_frames, first_frame, last_frame, filename_consistency, zero_byte, corrupt, resolution, bit_depth, alpha_channel, exr_type, timecode_plate_match, timecode_consistency, timecode_fps.

No oiiotool needed. Everything is read from the EXR header — one 64 KB read per sampled frame, no pixel decode. Full-sequence checks (count, gaps, names, sizes) touch every file; header checks sample first/middle/last plus every Nth frame (validation.sampleEvery).

Timecode is the primary check (spec §8.1a) and is compared against the plate, in this order: the timing plate named in the manifest → a plate found under plateSearchRoots → the API timecode minus handles, which is WARN-only because the API stores the editorial in-point and is a handle-length ahead of the plate by design.

Checksums are xxHash64 per file, with a sequence digest stored on the Export and reused by the delivery builder to verify what it packaged.

Server-side switches (SystemConfig, no worker redeploy): validation.enabled, validation.sampleEvery, validation.handleFrames, and validation.overrides for per-check severity, e.g. {"exr_type":"warn"}.

Blender renders

The worker also claims BLENDER_RENDER jobs: standalone renders submitted from the Blender add-on, with no Export behind them — nothing to validate, preview or QC. It runs blender -b file.blend -s N -e M -a, parses Fra:/Saved: output for progress, and reports complete or failed like any other job.

Set blenderPath in config.json to make a machine eligible. The version is reported at registration and the server only hands Blender jobs to machines that have it, so a workstation without Blender can never claim work it could only fail. A machine without it stays perfectly valid for After Effects work.

Blender exits 0 when a frame range is empty or an output path is unwritable, so a run that saves no frames is reported as a failure rather than a success.

See BlenderPanel/README.md.

Delivery builds

The same service also claims DELIVERY_BUILD jobs: creating a dated package folder, hard-linking (or copying) each shot's EXRs and delivery MOV into it, verifying every file against the validation-time checksum, and writing manifest.json plus the CSV report.

No After Effects involved, so these are claimed even while an artist has AE open — and at priority 10, inside the urgent threshold, so they run during the day rather than waiting for the render window. Someone is usually waiting on the folder.

deliveryRoot has to be reachable from this machine under the service account: --check verifies pathMappings targets, which is where a build most often falls over.

Full workflow: DELIVERY_WORKFLOW.md.

Checking a render by hand

.\VFXReviewWorker.exe --validate "V:\_EXPORTS\UNG\102\UNG_102_019_010\v005" --plate "V:\_FOOTAGE\UNG\UNG_102\UNG_102_019_010_BG01_TT_v001\UNG_102_019_010_BG01_TT_v001.01001.exr"

Runs the whole engine over an existing folder with no server, job or queue, and prints the report. The frame range, pattern and resolution are inferred from the files. Exits non-zero if any check fails, so it also works in a script. Useful for auditing work that predates the pipeline, and for proving the checks on real shots before turning the stage on.

Preview generation (Phase 4) plugs into this same service.

Packaging for another machine

On a machine with the .NET 8+ SDK (only the packaging machine needs it):

.\scripts\publish.ps1 -Zip

Produces publish\win-x64\ containing exactly two things: a self-contained VFXReviewWorker.exe (~67 MB, no .NET runtime needed on the target) and scripts\vfxr_build_preview.jsx, which the exe loads from its own folder at runtime. Keep them together — the script is deliberately not embedded in the exe so the slate/preview build can be patched without a rebuild. -Zip also writes VFXReviewWorker_<date>.zip for copying.

Installing on the target

1. Copy the folder to the render machine, e.g. C:\pipeline\VFXReviewWorker.

2. Create C:\ProgramData\VFXReviewWorker\config.json (next section).

3. Verify before installing the service:

   .\VFXReviewWorker.exe --check
   

   This resolves every path, confirms aerender.exe / AfterFX.com / the build script exist, checks each pathMappings target is reachable from that machine, and pings the server to confirm the API key is accepted. It exits non-zero if anything would stop the worker running, so a wrong path is caught here instead of in the logs an hour later.

4. Install the service (below), then confirm the machine appears on the web Pipeline → Machines page.

Run --check again under the service account (runas /user:STUDIO\svc-render) if the service will run as someone other than the logged-in artist — mapped drives and share permissions differ per account, and that is the most common cause of a worker that registers fine but cannot find footage.

Upgrading later: stop the service, replace the folder, start it again. Config lives in ProgramData and is untouched.

Configure

Create C:\ProgramData\VFXReviewWorker\config.json (§7.8):

{
  "serverUrl": "https://review.twotalesvfx.com",
  "apiKey": "<API_SECRET_KEY>",
  "machineName": "RENDER-01",
  "aerenderPath": "C:\\Program Files\\Adobe\\Adobe After Effects 2026\\Support Files\\aerender.exe",
  "aeVersion": "24.3",
  "ffmpegPath": "C:\\pipeline\\bin\\ffmpeg.exe",
  "blenderPath": "C:\\Program Files\\Blender Foundation\\Blender 4.2\\blender.exe",
  "plateSearchRoots": ["V:/_FOOTAGE/UNG"],
  "pathMappings": [
    { "from": "//SAN/", "to": "S:/" }
  ]
}

There is no workerVersion key: the version is stamped into the exe at build time and read back from the assembly, so a machine can never report a version it is not running. publish.ps1 appends the git commit, giving 1.1.0+feb1c7b. Bump <Version> in VFXReviewWorker.csproj when publishing a change; the worker reports it on startup and Pipeline → Machines flags any machine still on an older one.

pathMappings translate the manifest's canonical UNC paths to this machine's drive mappings. plateSearchRoots is optional: it is only consulted when a manifest carries no timing plate (older panel versions), to find the shot's plates for the timecode check. Everything tunable (poll interval, lease, stall timeout, max attempts, validation) lives in the server's SystemConfig and arrives at registration — no per-machine tuning files.

Run interactively (first-time smoke test)

VFXReviewWorker.exe

Logs go to the console-less service log at %ProgramData%\VFXReviewWorker\logs\worker_YYYYMMDD.log (14-day rolling). Confirm the machine appears on the web Pipeline → Machines page, then stop it.

Install as a Windows service

Run as a studio account with SAN access (works whether or not an artist is logged in):

sc.exe create VFXReviewRenderWorker binPath= "C:\pipeline\VFXReviewWorker\VFXReviewWorker.exe" start= auto obj= "STUDIO\svc-render" password= "<password>"
sc.exe description VFXReviewRenderWorker "VFXReview render pipeline worker (aerender)"
sc.exe start VFXReviewRenderWorker

Uninstall: sc.exe stop VFXReviewRenderWorker && sc.exe delete VFXReviewRenderWorker.

Rollout notes (spec §17.2)

• Install on one workstation first; add the second after a clean week.
• Set render.maxAttempts = 1 in SystemConfig for the first week (observe before auto-retrying).
• Enter each machine's render windows on the Machines page (availability), defaults: weekdays 19:00–08:00 + full weekends; Render Now override = 4 h.
• The dashboard warns when the two workstations report different AE versions.

Tests

cd RenderWorker
dotnet test

Covers the progress parser against recorded aerender transcript lines (success, error, non-retryable), ETA math, path mapping, log-tail ring buffer, and the output-dir clearing guard (only files matching the job's own pattern prefix are ever deleted).