vfxreview/RenderWorker/README.md

# VFXReview RenderWorker

Windows service that drives `aerender.exe` on the artist workstations (RenderPipeline2 §7).
It talks **only HTTP** to the VFXReview server (`/api/ext/*`, API-key auth) — it never
gets a database connection. All scheduling policy (render windows, Render Now,
urgent priority) is enforced server-side in the claim endpoint; the worker polls dumbly.

## One click → EXR + MOV + MP4

A single **Queue Export** click produces all three deliverables. The server
chains two jobs:

1. **AE_RENDER** — `aerender` writes the clean EXR sequence (no overlay, no LUT).
2. **PREVIEW_ONLY** — created automatically when the render completes. The
   worker runs `AfterFX.com -noui -r scripts/vfxr_build_preview.jsx`, which opens
   the studio slate/overlay **template** AEP, imports the rendered EXRs, rebuilds
   the shot around them (OCIO → `_SHOW LUT` → `UNG_VFX_OVERLAY`), duplicates
   `UNG_EXPORT_TEMPLATE` into a preview comp with the slate filled in, queues the
   MOV (`4444 Tri`) and MP4 (`REVIEW_PREVIEW`) output modules, and **saves a
   throwaway AEP**. The worker then runs `aerender -project <that aep>` with no
   `-comp`, rendering both outputs in one launch.

The MOV and MP4 land beside the EXRs. The MP4 is uploaded and registered as an
ordinary `Version` — internal-only, never client-visible, and it changes no task
or shot status (§10.0). Preview settings live in `SystemConfig` under
`preview.*`, so template names and paths are changed without redeploying.

If the preview stage fails, only it is retried — the validated EXRs are never
re-rendered. The temp AEP is kept on failure so you can open it and see exactly
what the farm built.

### Proving headless AE first

Headless AE is the one real unknown, so prove it before relying on it
(spec 18.2-C4). **Close After Effects**, then:

```powershell
.\scripts\test-preview-build.ps1 -ExrDir "V:\_EXPORTS\...\v002" -ShotCode "UNG_111_001_030" -Version "v002"
```

It runs the exact build the worker runs and prints the result plus any warnings
(missing LUT comp, missing slate layer, …), then gives you the `aerender` command
to render what it built. If it reports no result file, AE cannot script headlessly
under that account — fall back to having the panel pre-build the preview comp in
the artist's AEP (then it is pure `aerender`), or the ffmpeg engine.

## What it does (Phase 2 scope)

- Registers on startup (E6) and receives server-supplied tuning (poll/heartbeat/lease/stall).
- Heartbeats every 30 s (E7) — the heartbeat response is also the cancel channel.
- Claims one `AE_RENDER` job at a time (E8), runs `aerender.exe` with the manifest's
  comp/frame-range/templates, parses `PROGRESS:` lines, reports progress + ETA (E9,
  renews the lease).
- Stall watchdog: no progress for `stallTimeoutSeconds` (default 600) → kill process tree, retryable fail.
- Deterministic errors (missing footage / missing comp / unopenable project) → non-retryable fail (E10);
  transient errors auto-requeue server-side up to `maxAttempts`.
- On success: uploads the full aerender log via presign (E20), reports complete (E11).
- Crash recovery: `current-job.json` written on claim; on restart the worker asks the server
  what became of the job and reports a retryable fail if it was still ours. Partial renders
  are never resumed — the next attempt clears its own output files and re-renders.
- Durable reporting: complete/fail reports spool to disk and replay in order with backoff —
  rendering continues while the server is down.
- **Validation (Phase 3)**: after a successful render the worker checks the sequence it just
  produced and reports every result in one call (E12). See below.

## Validation stage

Runs on the worker straight after the render, while it still holds the job and the files are
local. The Export sits in `VALIDATING` until the report lands; the server decides what the
results mean (any FAIL → `VALIDATION_FAILED`, otherwise the preview stage is queued).

Checks: `file_count`, `missing_frames`, `duplicate_frames`, `first_frame`, `last_frame`,
`filename_consistency`, `zero_byte`, `corrupt`, `resolution`, `bit_depth`, `alpha_channel`,
`exr_type`, `timecode_plate_match`, `timecode_consistency`, `timecode_fps`.

**No oiiotool needed.** Everything is read from the EXR header — one 64 KB read per sampled
frame, no pixel decode. Full-sequence checks (count, gaps, names, sizes) touch every file;
header checks sample first/middle/last plus every Nth frame (`validation.sampleEvery`).

**Timecode is the primary check** (spec §8.1a) and is compared against *the plate*, in this
order: the timing plate named in the manifest → a plate found under `plateSearchRoots` →
the API timecode minus handles, which is **WARN-only** because the API stores the editorial
in-point and is a handle-length ahead of the plate by design.

Checksums are xxHash64 per file, with a sequence digest stored on the Export and reused by
the delivery builder to verify what it packaged.

Server-side switches (SystemConfig, no worker redeploy): `validation.enabled`,
`validation.sampleEvery`, `validation.handleFrames`, and `validation.overrides` for per-check
severity, e.g. `{"exr_type":"warn"}`.

## Blender renders

The worker also claims `BLENDER_RENDER` jobs: standalone renders submitted from
the Blender add-on, with no Export behind them — nothing to validate, preview or
QC. It runs `blender -b file.blend -s N -e M -a`, parses `Fra:`/`Saved:` output
for progress, and reports complete or failed like any other job.

Set `blenderPath` in `config.json` to make a machine eligible. The version is
reported at registration and **the server only hands Blender jobs to machines
that have it**, so a workstation without Blender can never claim work it could
only fail. A machine without it stays perfectly valid for After Effects work.

Blender exits 0 when a frame range is empty or an output path is unwritable, so
a run that saves no frames is reported as a failure rather than a success.

See [BlenderPanel/README.md](../BlenderPanel/README.md).

## Delivery builds

The same service also claims `DELIVERY_BUILD` jobs: creating a dated package
folder, hard-linking (or copying) each shot's EXRs and delivery MOV into it,
verifying every file against the validation-time checksum, and writing
`manifest.json` plus the CSV report.

No After Effects involved, so these are claimed **even while an artist has AE
open** — and at priority 10, inside the urgent threshold, so they run during the
day rather than waiting for the render window. Someone is usually waiting on the
folder.

`deliveryRoot` has to be reachable from this machine under the service account:
`--check` verifies `pathMappings` targets, which is where a build most often
falls over.

Full workflow: [DELIVERY_WORKFLOW.md](../DELIVERY_WORKFLOW.md).

### Checking a render by hand

```powershell
.\VFXReviewWorker.exe --validate "V:\_EXPORTS\UNG\102\UNG_102_019_010\v005" --plate "V:\_FOOTAGE\UNG\UNG_102\UNG_102_019_010_BG01_TT_v001\UNG_102_019_010_BG01_TT_v001.01001.exr"
```

Runs the whole engine over an existing folder with no server, job or queue, and prints the
report. The frame range, pattern and resolution are inferred from the files. Exits non-zero
if any check fails, so it also works in a script. Useful for auditing work that predates the
pipeline, and for proving the checks on real shots before turning the stage on.

Preview generation (Phase 4) plugs into this same service.

## Packaging for another machine

On a machine with the .NET 8+ SDK (only the packaging machine needs it):

```powershell
.\scripts\publish.ps1 -Zip
```

Produces `publish\win-x64\` containing exactly two things: a self-contained
`VFXReviewWorker.exe` (~67 MB, no .NET runtime needed on the target) and
`scripts\vfxr_build_preview.jsx`, which the exe loads from its own folder at
runtime. **Keep them together** — the script is deliberately not embedded in the
exe so the slate/preview build can be patched without a rebuild. `-Zip` also
writes `VFXReviewWorker_<date>.zip` for copying.

### Installing on the target

1. Copy the folder to the render machine, e.g. `C:\pipeline\VFXReviewWorker`.
2. Create `C:\ProgramData\VFXReviewWorker\config.json` (next section).
3. **Verify before installing the service:**

   ```powershell
   .\VFXReviewWorker.exe --check
   ```

   This resolves every path, confirms `aerender.exe` / `AfterFX.com` / the build
   script exist, checks each `pathMappings` target is reachable *from that
   machine*, and pings the server to confirm the API key is accepted. It exits
   non-zero if anything would stop the worker running, so a wrong path is caught
   here instead of in the logs an hour later.

4. Install the service (below), then confirm the machine appears on the web
   **Pipeline → Machines** page.

Run `--check` again under the *service account* (`runas /user:STUDIO\svc-render`)
if the service will run as someone other than the logged-in artist — mapped
drives and share permissions differ per account, and that is the most common
cause of a worker that registers fine but cannot find footage.

Upgrading later: stop the service, replace the folder, start it again. Config
lives in ProgramData and is untouched.

## Configure

Create `C:\ProgramData\VFXReviewWorker\config.json` (§7.8):

```json
{
  "serverUrl": "https://review.twotalesvfx.com",
  "apiKey": "<API_SECRET_KEY>",
  "machineName": "RENDER-01",
  "aerenderPath": "C:\\Program Files\\Adobe\\Adobe After Effects 2026\\Support Files\\aerender.exe",
  "aeVersion": "24.3",
  "ffmpegPath": "C:\\pipeline\\bin\\ffmpeg.exe",
  "blenderPath": "C:\\Program Files\\Blender Foundation\\Blender 4.2\\blender.exe",
  "plateSearchRoots": ["V:/_FOOTAGE/UNG"],
  "pathMappings": [
    { "from": "//SAN/", "to": "S:/" }
  ]
}
```

There is no `workerVersion` key: the version is stamped into the exe at build time and
read back from the assembly, so a machine can never report a version it is not running.
`publish.ps1` appends the git commit, giving `1.1.0+feb1c7b`. Bump `<Version>` in
`VFXReviewWorker.csproj` when publishing a change; the worker reports it on startup and
**Pipeline → Machines** flags any machine still on an older one.

`pathMappings` translate the manifest's canonical UNC paths to this machine's drive
mappings. `plateSearchRoots` is optional: it is only consulted when a manifest carries no
timing plate (older panel versions), to find the shot's plates for the timecode check.
Everything tunable (poll interval, lease, stall timeout, max attempts, validation) lives in
the server's SystemConfig and arrives at registration — no per-machine tuning files.

## Run interactively (first-time smoke test)

```bash
VFXReviewWorker.exe
```

Logs go to the console-less service log at `%ProgramData%\VFXReviewWorker\logs\worker_YYYYMMDD.log`
(14-day rolling). Confirm the machine appears on the web **Pipeline → Machines** page, then stop it.

## Install as a Windows service

Run as a studio account with SAN access (works whether or not an artist is logged in):

```powershell
sc.exe create VFXReviewRenderWorker binPath= "C:\pipeline\VFXReviewWorker\VFXReviewWorker.exe" start= auto obj= "STUDIO\svc-render" password= "<password>"
sc.exe description VFXReviewRenderWorker "VFXReview render pipeline worker (aerender)"
sc.exe start VFXReviewRenderWorker
```

Uninstall: `sc.exe stop VFXReviewRenderWorker && sc.exe delete VFXReviewRenderWorker`.

## Rollout notes (spec §17.2)

- Install on **one** workstation first; add the second after a clean week.
- Set `render.maxAttempts = 1` in SystemConfig for the first week (observe before auto-retrying).
- Enter each machine's render windows on the Machines page (`availability`), defaults:
  weekdays 19:00–08:00 + full weekends; Render Now override = 4 h.
- The dashboard warns when the two workstations report different AE versions.

## Tests

```bash
cd RenderWorker
dotnet test
```

Covers the progress parser against recorded aerender transcript lines (success, error,
non-retryable), ETA math, path mapping, log-tail ring buffer, and the output-dir
clearing guard (only files matching the job's own pattern prefix are ever deleted).