feat(pipeline): render queue, worker service and automated preview generation

One "Queue Export" click now renders the EXR sequence, then rebuilds the shot
headlessly with the studio slate/overlay template to produce the delivery MOV
and review MP4. Implements RenderPipeline2 phases 1-2 plus the preview stage.

Server:
- New models Export, RenderJob, ExportEvent, Machine, WorkerHeartbeat, plus
  Project.deliveryConfig and per-submission slate fields (Export.vfxScope,
  Export.submissionNote, inherited from the shot's previous export).
  Both migrations are purely additive; no existing column is touched.
- lib/render-pipeline: server-enforced state machine, transactional version
  increment with supersede, atomic FOR UPDATE SKIP LOCKED claim gated by
  machine availability windows, and a lease reaper run from instrumentation.ts.
- /api/ext/* endpoints for the panel and workers; session-auth mirrors under
  /api/render and /api/machines for the web UI.
- Pipeline pages: render queue, export detail, machine monitoring, plus an
  Exports tab on shot detail.

RenderWorker (.NET 8 Windows service, new):
- Registration, heartbeat as cancel channel, claim loop, aerender runner with
  progress parsing and stall watchdog, crash recovery and disk-spooled
  reporting that survives server downtime.
- Preview stage: headless AE assembles the preview comp into a throwaway AEP
  with both output modules queued, then a single aerender pass renders them.
  Preview jobs are not claimed while an interactive AE session is open, so an
  artist's project is never taken over.

AE panel: Queue Export with live status polling, urgent flag, retry, and the
VFX Scope / Submission Note fields. Every existing panel action is unchanged.

Preview chaining ships disabled behind SystemConfig preview.enabled.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
twotalesanimation
2026-08-02 14:34:34 +02:00
parent 0d0f3e1a33
commit ae58dc0366
71 changed files with 11767 additions and 1 deletions
+1380
View File
File diff suppressed because it is too large Load Diff
+4
View File
@@ -0,0 +1,4 @@
bin/
obj/
publish/
*.user
+137
View File
@@ -0,0 +1,137 @@
# VFXReview RenderWorker
Windows service that drives `aerender.exe` on the artist workstations (RenderPipeline2 §7).
It talks **only HTTP** to the VFXReview server (`/api/ext/*`, API-key auth) — it never
gets a database connection. All scheduling policy (render windows, Render Now,
urgent priority) is enforced server-side in the claim endpoint; the worker polls dumbly.
## One click → EXR + MOV + MP4
A single **Queue Export** click produces all three deliverables. The server
chains two jobs:
1. **AE_RENDER**`aerender` writes the clean EXR sequence (no overlay, no LUT).
2. **PREVIEW_ONLY** — created automatically when the render completes. The
worker runs `AfterFX.com -noui -r scripts/vfxr_build_preview.jsx`, which opens
the studio slate/overlay **template** AEP, imports the rendered EXRs, rebuilds
the shot around them (OCIO → `_SHOW LUT``UNG_VFX_OVERLAY`), duplicates
`UNG_EXPORT_TEMPLATE` into a preview comp with the slate filled in, queues the
MOV (`4444 Tri`) and MP4 (`REVIEW_PREVIEW`) output modules, and **saves a
throwaway AEP**. The worker then runs `aerender -project <that aep>` with no
`-comp`, rendering both outputs in one launch.
The MOV and MP4 land beside the EXRs. The MP4 is uploaded and registered as an
ordinary `Version` — internal-only, never client-visible, and it changes no task
or shot status (§10.0). Preview settings live in `SystemConfig` under
`preview.*`, so template names and paths are changed without redeploying.
If the preview stage fails, only it is retried — the validated EXRs are never
re-rendered. The temp AEP is kept on failure so you can open it and see exactly
what the farm built.
### Proving headless AE first
Headless AE is the one real unknown, so prove it before relying on it
(spec 18.2-C4). **Close After Effects**, then:
```powershell
.\scripts\test-preview-build.ps1 -ExrDir "V:\_EXPORTS\...\v002" -ShotCode "UNG_111_001_030" -Version "v002"
```
It runs the exact build the worker runs and prints the result plus any warnings
(missing LUT comp, missing slate layer, …), then gives you the `aerender` command
to render what it built. If it reports no result file, AE cannot script headlessly
under that account — fall back to having the panel pre-build the preview comp in
the artist's AEP (then it is pure `aerender`), or the ffmpeg engine.
## What it does (Phase 2 scope)
- Registers on startup (E6) and receives server-supplied tuning (poll/heartbeat/lease/stall).
- Heartbeats every 30 s (E7) — the heartbeat response is also the cancel channel.
- Claims one `AE_RENDER` job at a time (E8), runs `aerender.exe` with the manifest's
comp/frame-range/templates, parses `PROGRESS:` lines, reports progress + ETA (E9,
renews the lease).
- Stall watchdog: no progress for `stallTimeoutSeconds` (default 600) → kill process tree, retryable fail.
- Deterministic errors (missing footage / missing comp / unopenable project) → non-retryable fail (E10);
transient errors auto-requeue server-side up to `maxAttempts`.
- On success: uploads the full aerender log via presign (E20), reports complete (E11).
- Crash recovery: `current-job.json` written on claim; on restart the worker asks the server
what became of the job and reports a retryable fail if it was still ours. Partial renders
are never resumed — the next attempt clears its own output files and re-renders.
- Durable reporting: complete/fail reports spool to disk and replay in order with backoff —
rendering continues while the server is down.
Preview generation (Phase 4) and validation (Phase 3) plug into this same service later.
## Build
Requires the .NET 8+ SDK.
```bash
cd RenderWorker/VFXReviewWorker
dotnet publish -c Release -r win-x64 --self-contained -p:PublishSingleFile=true -o publish
```
Produces a single `publish/VFXReviewWorker.exe` — no runtime install needed on render nodes.
## Configure
Create `C:\ProgramData\VFXReviewWorker\config.json` (§7.8):
```json
{
"serverUrl": "https://review.twotalesvfx.com",
"apiKey": "<API_SECRET_KEY>",
"machineName": "RENDER-01",
"aerenderPath": "C:\\Program Files\\Adobe\\Adobe After Effects 2026\\Support Files\\aerender.exe",
"aeVersion": "24.3",
"ffmpegPath": "C:\\pipeline\\bin\\ffmpeg.exe",
"pathMappings": [
{ "from": "//SAN/", "to": "S:/" }
]
}
```
`pathMappings` translate the manifest's canonical UNC paths to this machine's drive
mappings. Everything tunable (poll interval, lease, stall timeout, max attempts) lives in
the server's SystemConfig and arrives at registration — no per-machine tuning files.
## Run interactively (first-time smoke test)
```bash
VFXReviewWorker.exe
```
Logs go to the console-less service log at `%ProgramData%\VFXReviewWorker\logs\worker_YYYYMMDD.log`
(14-day rolling). Confirm the machine appears on the web **Pipeline → Machines** page, then stop it.
## Install as a Windows service
Run as a studio account with SAN access (works whether or not an artist is logged in):
```powershell
sc.exe create VFXReviewRenderWorker binPath= "C:\pipeline\VFXReviewWorker\VFXReviewWorker.exe" start= auto obj= "STUDIO\svc-render" password= "<password>"
sc.exe description VFXReviewRenderWorker "VFXReview render pipeline worker (aerender)"
sc.exe start VFXReviewRenderWorker
```
Uninstall: `sc.exe stop VFXReviewRenderWorker && sc.exe delete VFXReviewRenderWorker`.
## Rollout notes (spec §17.2)
- Install on **one** workstation first; add the second after a clean week.
- Set `render.maxAttempts = 1` in SystemConfig for the first week (observe before auto-retrying).
- Enter each machine's render windows on the Machines page (`availability`), defaults:
weekdays 19:0008:00 + full weekends; Render Now override = 4 h.
- The dashboard warns when the two workstations report different AE versions.
## Tests
```bash
cd RenderWorker
dotnet test
```
Covers the progress parser against recorded aerender transcript lines (success, error,
non-retryable), ETA math, path mapping, log-tail ring buffer, and the output-dir
clearing guard (only files matching the job's own pattern prefix are ever deleted).
@@ -0,0 +1,87 @@
using VFXReviewWorker;
using Xunit;
namespace VFXReviewWorker.Tests;
public class PathMapperTests
{
private static PathMapper Mapper(params (string from, string to)[] maps) =>
new(maps.Select(m => new PathMapping { From = m.from, To = m.to }));
[Fact]
public void MapsUncToDriveLetter()
{
var mapper = Mapper(("//SAN/", "S:/"));
Assert.Equal(@"S:\renders\UNG_S1\106\shot\v004", mapper.Map("//SAN/renders/UNG_S1/106/shot/v004"));
}
[Fact]
public void MapsBackslashInputToo()
{
var mapper = Mapper(("//SAN/", "S:/"));
Assert.Equal(@"S:\projects\a.aep", mapper.Map(@"\\SAN\projects\a.aep"));
}
[Fact]
public void CaseInsensitiveMatch()
{
var mapper = Mapper(("//SAN/", "S:/"));
Assert.Equal(@"S:\x", mapper.Map("//san/x"));
}
[Fact]
public void FirstMatchingMappingWins()
{
var mapper = Mapper(("//SAN/renders/", "R:/"), ("//SAN/", "S:/"));
Assert.Equal(@"R:\a", mapper.Map("//SAN/renders/a"));
Assert.Equal(@"S:\projects\a", mapper.Map("//SAN/projects/a"));
}
[Fact]
public void UnmappedPathPassesThroughWithLocalSeparators()
{
var mapper = Mapper(("//SAN/", "S:/"));
Assert.Equal(@"D:\local\thing", mapper.Map("D:/local/thing"));
}
}
public class PrepareOutputDirTests
{
[Fact]
public void ClearsOnlyFilesMatchingPatternPrefix()
{
var dir = Path.Combine(Path.GetTempPath(), "vfxr-test-" + Guid.NewGuid().ToString("N"));
Directory.CreateDirectory(dir);
try
{
File.WriteAllText(Path.Combine(dir, "SHOT_cmp_TT_v004.1001.exr"), "old");
File.WriteAllText(Path.Combine(dir, "SHOT_cmp_TT_v004.1002.exr"), "old");
File.WriteAllText(Path.Combine(dir, "unrelated-notes.txt"), "keep me");
AerenderRunner.PrepareOutputDir(dir, "SHOT_cmp_TT_v004.[####].exr");
Assert.False(File.Exists(Path.Combine(dir, "SHOT_cmp_TT_v004.1001.exr")));
Assert.False(File.Exists(Path.Combine(dir, "SHOT_cmp_TT_v004.1002.exr")));
Assert.True(File.Exists(Path.Combine(dir, "unrelated-notes.txt")));
}
finally
{
Directory.Delete(dir, recursive: true);
}
}
[Fact]
public void CreatesMissingDirectory()
{
var dir = Path.Combine(Path.GetTempPath(), "vfxr-test-" + Guid.NewGuid().ToString("N"), "v001");
try
{
AerenderRunner.PrepareOutputDir(dir, "X.[####].exr");
Assert.True(Directory.Exists(dir));
}
finally
{
Directory.Delete(Path.GetDirectoryName(dir)!, recursive: true);
}
}
}
@@ -0,0 +1,120 @@
using VFXReviewWorker;
using Xunit;
namespace VFXReviewWorker.Tests;
/// <summary>
/// Recorded-transcript tests (§17.2): lines below are representative aerender
/// stdout from AE 2024/2025/2026 successes, errors, and stalls.
/// </summary>
public class ProgressParserTests
{
[Theory]
[InlineData("PROGRESS: 0:00:02:03 (51): 0 Seconds", 51)]
[InlineData("PROGRESS: 2 (2): 1 Seconds", 2)]
[InlineData("PROGRESS: 0:00:00:00 (1): 12 Seconds", 1)]
[InlineData("PROGRESS: 0:00:04:23 (120): 3 Seconds", 120)]
public void ParsesProgressFrames(string line, int expected)
{
Assert.Equal(expected, ProgressParser.ParseFrame(line));
}
[Theory]
[InlineData("PROGRESS: Rendering frame data")]
[InlineData("Starting composition \"UNG_106_010_020_cmp\".")]
[InlineData("aerender version 24.3x52")]
[InlineData("")]
public void IgnoresNonFrameLines(string line)
{
Assert.Null(ProgressParser.ParseFrame(line));
}
[Theory]
[InlineData("aerender ERROR: An existing connection was forcibly closed")]
[InlineData("aerender ERROR -1610153464: After Effects error")]
[InlineData("ERROR: Unable to open project")]
[InlineData("aerender SYNTAX ERROR: Illegal argument flag: false")]
public void DetectsErrorLines(string line)
{
Assert.True(ProgressParser.IsErrorLine(line));
}
[Fact]
public void ProgressLineIsNotAnError()
{
Assert.False(ProgressParser.IsErrorLine("PROGRESS: 0:00:02:03 (51): 0 Seconds"));
}
[Theory]
[InlineData("aerender ERROR: After Effects error: layer source file is missing or inaccessible")]
[InlineData("aerender ERROR: No comp was found with the given name.")]
[InlineData("aerender ERROR: project file could not be opened")]
[InlineData("WARNING: Missing footage in comp")]
[InlineData("aerender SYNTAX ERROR: Illegal argument flag: false")]
public void DeterministicErrorsAreNonRetryable(string line)
{
Assert.True(ProgressParser.IsNonRetryableError(line));
}
[Theory]
[InlineData("aerender ERROR: An existing connection was forcibly closed")]
[InlineData("aerender ERROR -1610153464: After Effects crashed")]
public void TransientErrorsStayRetryable(string line)
{
Assert.False(ProgressParser.IsNonRetryableError(line));
}
}
public class EtaCalculatorTests
{
[Fact]
public void NoFramesMeansNoEstimate()
{
var eta = new EtaCalculator();
Assert.Null(eta.EstimateSeconds(100));
}
[Fact]
public void EstimatesFromRollingAverage()
{
var eta = new EtaCalculator();
var t = DateTimeOffset.UtcNow;
eta.RecordFrame(t);
eta.RecordFrame(t.AddSeconds(2));
eta.RecordFrame(t.AddSeconds(4)); // 2 s/frame average
Assert.Equal(20, eta.EstimateSeconds(10));
}
[Fact]
public void ZeroRemainingMeansNoEstimate()
{
var eta = new EtaCalculator();
var t = DateTimeOffset.UtcNow;
eta.RecordFrame(t);
eta.RecordFrame(t.AddSeconds(1));
Assert.Null(eta.EstimateSeconds(0));
}
[Fact]
public void RollingWindowDropsOldFrames()
{
var eta = new EtaCalculator(windowSize: 2);
var t = DateTimeOffset.UtcNow;
eta.RecordFrame(t);
eta.RecordFrame(t.AddSeconds(100)); // old slow frame, will roll out
eta.RecordFrame(t.AddSeconds(101));
eta.RecordFrame(t.AddSeconds(102)); // window now [1, 1]
Assert.Equal(10, eta.EstimateSeconds(10));
}
}
public class LogTailTests
{
[Fact]
public void KeepsOnlyLastNLines()
{
var tail = new LogTail(3);
for (var i = 1; i <= 5; i++) tail.Add($"line {i}");
Assert.Equal("line 3\nline 4\nline 5", tail.ToString());
}
}
@@ -0,0 +1,20 @@
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<TargetFramework>net8.0</TargetFramework>
<Nullable>enable</Nullable>
<ImplicitUsings>enable</ImplicitUsings>
<IsPackable>false</IsPackable>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.11.1" />
<PackageReference Include="xunit" Version="2.9.2" />
<PackageReference Include="xunit.runner.visualstudio" Version="2.8.2" />
</ItemGroup>
<ItemGroup>
<ProjectReference Include="..\VFXReviewWorker\VFXReviewWorker.csproj" />
</ItemGroup>
</Project>
+33
View File
@@ -0,0 +1,33 @@
using System.Diagnostics;
namespace VFXReviewWorker;
/// <summary>
/// Detects an *interactive* After Effects session on this machine.
///
/// This matters because `AfterFX.com -noui -r script.jsx` is handed to an
/// already-running After Effects instance rather than starting an isolated
/// one — so running the preview build while an artist has AE open would take
/// over their session and quit it, losing unsaved work.
///
/// Deliberately conservative: this matches any After Effects process, the
/// interactive `AfterFX.exe` as well as the `AfterFX.com` render engine that
/// aerender drives. Waiting for either to finish also avoids two AE instances
/// competing for the same box, and the worst case is only that a preview build
/// starts a little later.
/// </summary>
public static class AeSession
{
public static bool InteractiveRunning()
{
try
{
return Process.GetProcessesByName("AfterFX").Length > 0;
}
catch
{
// If we cannot tell, assume an artist is working — never risk their session.
return true;
}
}
}
@@ -0,0 +1,298 @@
using System.Diagnostics;
using System.Text;
using Microsoft.Extensions.Logging;
namespace VFXReviewWorker;
public sealed record RenderResult(
bool Success,
int ExitCode,
bool Retryable,
string? ErrorMessage,
string LogTail,
string FullLogPath,
double RenderSeconds);
/// <summary>
/// Stage 1: aerender execution (§7.3). Launches aerender.exe, parses stdout
/// for PROGRESS/ERROR lines, reports progress (renewing the lease), enforces
/// the stall timeout, and honours cancellation by killing the process tree.
/// </summary>
public sealed class AerenderRunner
{
private readonly WorkerOptions _options;
private readonly ApiClient _api;
private readonly ILogger<AerenderRunner> _log;
public AerenderRunner(WorkerOptions options, ApiClient api, ILogger<AerenderRunner> log)
{
_options = options;
_api = api;
_log = log;
}
/// <summary>
/// Renders every queued item in an already-prepared project (no -comp), used
/// by the preview stage: the build script saved an AEP with the MOV and MP4
/// output modules already queued, so one launch produces both.
/// </summary>
public Task<RenderResult> RunProjectAsync(
string jobId,
string machineId,
string projectPathLocal,
int totalFrames,
ServerConfig config,
CancellationToken cancelJob,
CancellationToken shutdown)
{
var args = new List<string> { "-project", projectPathLocal, "-mp" };
_log.LogInformation("Rendering prepared project for job {JobId}: {Project}", jobId, projectPathLocal);
return ExecuteAsync(jobId, machineId, args, totalFrames, 0, config, $"aerender_{jobId}_preview.log", cancelJob, shutdown);
}
public async Task<RenderResult> RunAsync(
ClaimedJob job,
RenderManifest m,
string machineId,
string outputDirLocal,
ServerConfig config,
CancellationToken cancelJob,
CancellationToken shutdown)
{
var mapper = new PathMapper(_options.PathMappings);
var aepLocal = mapper.Map(m.AepPath);
var outputArg = Path.Combine(outputDirLocal, m.OutputPattern);
PrepareOutputDir(outputDirLocal, m.OutputPattern);
var args = new List<string>
{
"-project", aepLocal,
"-comp", m.CompName,
"-s", m.FrameStart.ToString(),
"-e", m.FrameEnd.ToString(),
};
if (!string.IsNullOrEmpty(m.RenderSettingsTemplate)) { args.Add("-RStemplate"); args.Add(m.RenderSettingsTemplate); }
if (!string.IsNullOrEmpty(m.OutputModuleTemplate)) { args.Add("-OMtemplate"); args.Add(m.OutputModuleTemplate); }
args.Add("-output"); args.Add(outputArg);
// NB: no "-continueOnMissingFootage" — the real flag takes no value
// (passing "false" is an aerender SYNTAX ERROR), and its *presence*
// enables skipping missing footage. aerender's default is to stop on
// missing footage, which is exactly what the pipeline wants.
_log.LogInformation("Launching aerender for job {JobId}: {Aep} comp \"{Comp}\" frames {S}-{E} → {Out}",
job.Id, aepLocal, m.CompName, m.FrameStart, m.FrameEnd, outputArg);
return await ExecuteAsync(job.Id, machineId, args, m.TotalFrames, m.FrameStart, config,
$"aerender_{job.Id}.log", cancelJob, shutdown);
}
private async Task<RenderResult> ExecuteAsync(
string jobId,
string machineId,
List<string> args,
int totalFrames,
int frameStart,
ServerConfig config,
string logFileName,
CancellationToken cancelJob,
CancellationToken shutdown)
{
var fullLogPath = Path.Combine(WorkerOptions.LogsDir, logFileName);
Directory.CreateDirectory(WorkerOptions.LogsDir);
var psi = new ProcessStartInfo
{
FileName = _options.AerenderPath,
RedirectStandardOutput = true,
RedirectStandardError = true,
UseShellExecute = false,
CreateNoWindow = true,
StandardOutputEncoding = Encoding.UTF8,
StandardErrorEncoding = Encoding.UTF8,
};
foreach (var a in args) psi.ArgumentList.Add(a);
var tail = new LogTail(200);
var eta = new EtaCalculator();
var sw = Stopwatch.StartNew();
var lastProgressAt = DateTimeOffset.UtcNow;
var lastReportAt = DateTimeOffset.MinValue;
int framesDone = 0;
string? firstError = null;
bool nonRetryable = false;
bool cancelledByServer = false;
using var process = new Process { StartInfo = psi };
await using var fullLog = new StreamWriter(fullLogPath, append: false, Encoding.UTF8);
var logLock = new object();
void HandleLine(string? line, bool isStderr)
{
if (line is null) return;
lock (logLock)
{
fullLog.WriteLine(line);
tail.Add(line);
}
var frame = ProgressParser.ParseFrame(line);
if (frame is not null)
{
framesDone = Math.Max(framesDone, frame.Value);
lastProgressAt = DateTimeOffset.UtcNow;
eta.RecordFrame(lastProgressAt);
}
if ((isStderr || ProgressParser.IsErrorLine(line)) && !string.IsNullOrWhiteSpace(line))
{
if (ProgressParser.IsErrorLine(line)) firstError ??= line.Trim();
if (ProgressParser.IsNonRetryableError(line)) nonRetryable = true;
}
}
process.OutputDataReceived += (_, e) => HandleLine(e.Data, isStderr: false);
process.ErrorDataReceived += (_, e) => HandleLine(e.Data, isStderr: true);
try
{
process.Start();
}
catch (Exception ex)
{
return new RenderResult(false, -1, Retryable: false,
$"Failed to launch aerender at {_options.AerenderPath}: {ex.Message}",
tail.ToString(), fullLogPath, 0);
}
process.BeginOutputReadLine();
process.BeginErrorReadLine();
var total = totalFrames;
while (!process.HasExited)
{
await Task.Delay(TimeSpan.FromSeconds(2), CancellationToken.None);
if (cancelJob.IsCancellationRequested || shutdown.IsCancellationRequested)
{
_log.LogWarning("Cancellation requested — killing aerender tree for job {JobId}", jobId);
KillTree(process);
cancelledByServer = cancelJob.IsCancellationRequested;
break;
}
// Stall watchdog (§7.3): no progress line for stallTimeoutSeconds → kill
if ((DateTimeOffset.UtcNow - lastProgressAt).TotalSeconds > config.StallTimeoutSeconds)
{
_log.LogError("aerender stalled ({Sec}s without progress) — killing job {JobId}", config.StallTimeoutSeconds, jobId);
KillTree(process);
await WaitForExitSafe(process);
return new RenderResult(false, -2, Retryable: true,
$"aerender produced no progress for {config.StallTimeoutSeconds}s (stall timeout)",
tail.ToString(), fullLogPath, sw.Elapsed.TotalSeconds);
}
// Progress report every ~10 s (renews lease); best-effort
if ((DateTimeOffset.UtcNow - lastReportAt).TotalSeconds >= 10)
{
lastReportAt = DateTimeOffset.UtcNow;
var progress = total > 0 ? Math.Min(1.0, (double)framesDone / total) : 0;
try
{
var resp = await _api.ProgressAsync(jobId, machineId, progress,
framesDone > 0 ? frameStart + framesDone - 1 : null,
total, eta.EstimateSeconds(total - framesDone), tail.ToString(), shutdown);
if (resp?.CancelRequested == true)
{
_log.LogWarning("Server requested cancel for job {JobId}", jobId);
KillTree(process);
cancelledByServer = true;
break;
}
}
catch (Exception ex)
{
_log.LogDebug(ex, "Progress report failed (non-fatal; lease covered by reaper)");
}
}
}
await WaitForExitSafe(process);
sw.Stop();
lock (logLock) { fullLog.Flush(); }
if (cancelledByServer)
{
return new RenderResult(false, -3, Retryable: false, "Cancelled", tail.ToString(), fullLogPath, sw.Elapsed.TotalSeconds);
}
var exitCode = process.ExitCode;
if (exitCode == 0 && firstError is null)
{
return new RenderResult(true, 0, true, null, tail.ToString(), fullLogPath, sw.Elapsed.TotalSeconds);
}
return new RenderResult(false, exitCode, Retryable: !nonRetryable,
firstError ?? $"aerender exited with code {exitCode}",
tail.ToString(), fullLogPath, sw.Elapsed.TotalSeconds);
}
/// <summary>
/// §7.3: outputDir is created before launch; a pre-existing non-empty dir
/// can only hold output from a previous crashed attempt of this same
/// immutable version. Safety guard: only files matching the job's own
/// output pattern prefix are ever deleted.
/// </summary>
/// <summary>Filename prefix of an output pattern ("X.[####].exr" → "X").</summary>
internal static string OutputPatternPrefix(string outputPattern)
{
var bracket = outputPattern.IndexOf('[');
return bracket > 0 ? outputPattern[..bracket].TrimEnd('.') : Path.GetFileNameWithoutExtension(outputPattern);
}
internal static void PrepareOutputDir(string outputDirLocal, string outputPattern)
{
Directory.CreateDirectory(outputDirLocal);
var prefix = OutputPatternPrefix(outputPattern);
if (string.IsNullOrEmpty(prefix)) return;
foreach (var file in Directory.EnumerateFiles(outputDirLocal))
{
if (Path.GetFileName(file).StartsWith(prefix, StringComparison.OrdinalIgnoreCase))
{
try { File.Delete(file); } catch { /* locked leftovers surface as an aerender error */ }
}
}
}
private static void KillTree(Process process)
{
try
{
if (!process.HasExited) process.Kill(entireProcessTree: true);
}
catch { /* already gone */ }
}
private static async Task WaitForExitSafe(Process process)
{
try { await process.WaitForExitAsync(new CancellationTokenSource(TimeSpan.FromSeconds(30)).Token); }
catch { try { process.Kill(entireProcessTree: true); } catch { } }
}
}
/// <summary>Ring buffer of the last N output lines (§5.3 logTail).</summary>
public sealed class LogTail
{
private readonly Queue<string> _lines;
private readonly int _capacity;
public LogTail(int capacity)
{
_capacity = capacity;
_lines = new Queue<string>(capacity);
}
public void Add(string line)
{
_lines.Enqueue(line);
while (_lines.Count > _capacity) _lines.Dequeue();
}
public override string ToString() => string.Join('\n', _lines);
}
+184
View File
@@ -0,0 +1,184 @@
using System.Net;
using System.Net.Http.Headers;
using System.Net.Http.Json;
using System.Text.Json;
using Microsoft.Extensions.Logging;
namespace VFXReviewWorker;
/// <summary>
/// HTTP client for the VFXReview /api/ext pipeline endpoints. Workers talk
/// only HTTP — never a database (§2.1 principle 4).
/// </summary>
public sealed class ApiClient
{
private readonly HttpClient _http;
private readonly ILogger<ApiClient> _log;
public ApiClient(WorkerOptions options, ILogger<ApiClient> log)
{
_log = log;
_http = new HttpClient
{
BaseAddress = new Uri(options.ServerUrl.TrimEnd('/') + "/"),
Timeout = TimeSpan.FromSeconds(30),
};
_http.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", options.ApiKey);
}
private static readonly JsonSerializerOptions Json = WorkerOptions.JsonOpts;
// E6
public async Task<RegisterResponse> RegisterAsync(WorkerOptions o, CancellationToken ct)
{
var res = await _http.PostAsJsonAsync("api/ext/workers/register", new
{
name = o.MachineName,
hostname = Dns.GetHostName(),
workerVersion = o.WorkerVersion,
aeVersion = o.AeVersion,
capabilities = new { maxConcurrentJobs = 1, tools = new { ffmpeg = o.FfmpegPath != null, oiiotool = o.OiiotoolPath != null } },
}, Json, ct);
res.EnsureSuccessStatusCode();
return (await res.Content.ReadFromJsonAsync<RegisterResponse>(Json, ct))!;
}
// E7
public async Task<HeartbeatResponse> HeartbeatAsync(string machineId, string? currentJobId, CancellationToken ct)
{
var res = await _http.PostAsJsonAsync($"api/ext/workers/{machineId}/heartbeat", new
{
cpuPercent = (double?)null,
memPercent = (double?)null,
diskFreeGb = GetDiskFreeGb(),
currentJobId,
}, Json, ct);
res.EnsureSuccessStatusCode();
return (await res.Content.ReadFromJsonAsync<HeartbeatResponse>(Json, ct))!;
}
// E8 — null when the queue is empty / machine outside its render window (204)
public async Task<ClaimedJob?> ClaimAsync(string machineId, string[] types, CancellationToken ct)
{
var res = await _http.PostAsJsonAsync("api/ext/render/jobs/claim", new
{
machineId,
types,
}, Json, ct);
if (res.StatusCode == HttpStatusCode.NoContent) return null;
res.EnsureSuccessStatusCode();
var body = await res.Content.ReadFromJsonAsync<ClaimResponse>(Json, ct);
return body?.Job;
}
// E9 — best-effort; failures are swallowed by the caller (lease/reaper covers us)
public async Task<ProgressResponse?> ProgressAsync(string jobId, string machineId, double progress,
int? currentFrame, int? totalFrames, int? etaSeconds, string? logTail, CancellationToken ct)
{
using var req = new HttpRequestMessage(HttpMethod.Patch, $"api/ext/render/jobs/{jobId}/progress")
{
Content = JsonContent.Create(new { machineId, progress, currentFrame, totalFrames, etaSeconds, logTail }, options: Json),
};
var res = await _http.SendAsync(req, ct);
if (!res.IsSuccessStatusCode) return null;
return await res.Content.ReadFromJsonAsync<ProgressResponse>(Json, ct);
}
// E10 / E11 — lifecycle reports, sent via the spooler for durability.
// Returns true when the server accepted (2xx) OR permanently rejected (409
// contradictory / 404 gone) — both mean "stop retrying".
public async Task<bool> SendLifecycleAsync(string kind, string jobId, Dictionary<string, object?> payload, CancellationToken ct)
{
var path = kind switch
{
"fail" => $"api/ext/render/jobs/{jobId}/fail",
"complete-render" => $"api/ext/render/jobs/{jobId}/complete-render",
"finalize" => $"api/ext/render/jobs/{jobId}/finalize",
_ => throw new ArgumentOutOfRangeException(nameof(kind), kind, "unknown lifecycle report kind"),
};
var res = await _http.PostAsJsonAsync(path, payload, Json, ct);
if (res.IsSuccessStatusCode) return true;
if (res.StatusCode is HttpStatusCode.Conflict or HttpStatusCode.NotFound)
{
_log.LogWarning("Server permanently rejected {Kind} for job {JobId}: {Status} {Body}",
kind, jobId, (int)res.StatusCode, await res.Content.ReadAsStringAsync(ct));
return true; // don't retry contradictory/vanished reports
}
_log.LogWarning("Lifecycle report {Kind} for {JobId} failed with {Status}; will retry", kind, jobId, (int)res.StatusCode);
return false;
}
// E20 + PUT upload. Returns the storage key, or null when the upload failed.
public async Task<string?> UploadArtifactAsync(
string jobId, string machineId, string kind, string filePath, string contentType, CancellationToken ct)
{
try
{
if (!File.Exists(filePath))
{
_log.LogWarning("Artifact {Kind} not found at {Path}", kind, filePath);
return null;
}
var presignRes = await _http.PostAsJsonAsync($"api/ext/render/jobs/{jobId}/artifact-presign", new
{
machineId,
kind,
fileName = Path.GetFileName(filePath),
contentType,
}, Json, ct);
if (!presignRes.IsSuccessStatusCode)
{
_log.LogWarning("Presign for {Kind} failed: {Status}", kind, (int)presignRes.StatusCode);
return null;
}
var presign = await presignRes.Content.ReadFromJsonAsync<PresignResponse>(Json, ct);
if (presign is null) return null;
// Media files can be large — generous timeout, separate client so the
// presigned URL is not sent with our Authorization header.
using var upload = new HttpClient { Timeout = TimeSpan.FromMinutes(30) };
await using var stream = File.OpenRead(filePath);
var content = new StreamContent(stream);
content.Headers.ContentType = new MediaTypeHeaderValue(contentType);
var putRes = await upload.PutAsync(presign.PresignedUrl, content, ct);
if (!putRes.IsSuccessStatusCode)
{
_log.LogWarning("Upload of {Kind} failed: {Status}", kind, (int)putRes.StatusCode);
return null;
}
return presign.Key;
}
catch (Exception ex)
{
_log.LogWarning(ex, "Artifact upload ({Kind}) failed for job {JobId}", kind, jobId);
return null;
}
}
public Task<string?> UploadLogAsync(string jobId, string machineId, string logFilePath, CancellationToken ct) =>
UploadArtifactAsync(jobId, machineId, "log", logFilePath, "text/plain", ct);
// E3 — used by crash-recovery reconcile (§7.5)
public async Task<ExportDetail?> GetExportAsync(string exportId, CancellationToken ct)
{
var res = await _http.GetAsync($"api/ext/exports/{exportId}", ct);
if (!res.IsSuccessStatusCode) return null;
var body = await res.Content.ReadFromJsonAsync<ExportDetailResponse>(Json, ct);
return body?.Export;
}
private static double? GetDiskFreeGb()
{
try
{
var root = Path.GetPathRoot(Environment.SystemDirectory);
if (root is null) return null;
return Math.Round(new DriveInfo(root).AvailableFreeSpace / 1_073_741_824.0, 1);
}
catch
{
return null;
}
}
}
+189
View File
@@ -0,0 +1,189 @@
using System.Text.Json;
using System.Text.Json.Serialization;
namespace VFXReviewWorker;
// Wire types for the /api/ext pipeline endpoints (RenderPipeline2 §6).
public sealed class RegisterResponse
{
public MachineInfo Machine { get; set; } = new();
public ServerConfig Config { get; set; } = new();
}
public sealed class MachineInfo
{
public string Id { get; set; } = "";
public string Name { get; set; } = "";
public bool Enabled { get; set; }
}
/// <summary>Server-supplied tuning (E6) — fleet-tunable without touching installs.</summary>
public sealed class ServerConfig
{
public int PollSeconds { get; set; } = 10;
public int HeartbeatSeconds { get; set; } = 30;
public int LeaseSeconds { get; set; } = 300;
public int MaxAttempts { get; set; } = 3;
public int StallTimeoutSeconds { get; set; } = 600;
}
public sealed class ClaimResponse
{
public ClaimedJob Job { get; set; } = new();
}
public sealed class ClaimedJob
{
public string Id { get; set; } = "";
public string Type { get; set; } = "AE_RENDER";
public int Attempt { get; set; }
public int MaxAttempts { get; set; }
public string? ExportId { get; set; }
public DateTimeOffset? LeaseExpiresAt { get; set; }
/// <summary>Manifest shape depends on the job type — deserialize per stage.</summary>
public JsonElement Manifest { get; set; }
public RenderManifest AsRenderManifest() =>
Manifest.Deserialize<RenderManifest>(WorkerOptions.JsonOpts)
?? throw new InvalidOperationException("Job manifest is not a render manifest");
public PreviewManifest AsPreviewManifest() =>
Manifest.Deserialize<PreviewManifest>(WorkerOptions.JsonOpts)
?? throw new InvalidOperationException("Job manifest is not a preview manifest");
}
/// <summary>PREVIEW_ONLY manifest (server-built, RenderPipeline2 §9).</summary>
public sealed class PreviewManifest
{
public string Stage { get; set; } = "PREVIEW";
public string ExportId { get; set; } = "";
public string ShotCode { get; set; } = "";
public string VersionString { get; set; } = "";
public string OutputDir { get; set; } = "";
public string OutputPattern { get; set; } = "";
public int FrameStart { get; set; }
public int FrameEnd { get; set; }
public double Fps { get; set; } = 24;
public string TemplateAep { get; set; } = "";
public string TemplateComp { get; set; } = "";
public string? OverlayComp { get; set; }
public string? LutComp { get; set; }
public string MovTemplate { get; set; } = "";
public string Mp4Template { get; set; } = "";
public string MovOutput { get; set; } = "";
public string Mp4Output { get; set; } = "";
public string? SlateScopeProp { get; set; }
public string? SlateSubmissionProp { get; set; }
public SlateInfo Slate { get; set; } = new();
public int TotalFrames => FrameEnd - FrameStart + 1;
}
public sealed class SlateInfo
{
public string VersionName { get; set; } = "";
public string Date { get; set; } = "";
public string? Description { get; set; }
public string? Notes { get; set; }
public string ShotCode { get; set; } = "";
public string? Episode { get; set; }
public string? Scene { get; set; }
public string? VfxScope { get; set; }
public string? SubmissionNote { get; set; }
}
/// <summary>Result JSON written by vfxr_build_preview.jsx.</summary>
public sealed class PreviewBuildResult
{
public bool Ok { get; set; }
public string? Error { get; set; }
public List<string> Warnings { get; set; } = new();
public string? TempAep { get; set; }
public string? PreviewComp { get; set; }
public int? FrameCount { get; set; }
public int? Width { get; set; }
public int? Height { get; set; }
}
/// <summary>The Render Manifest (§6.2) — only the fields the Phase 2 render stage needs; the rest is preserved raw.</summary>
public sealed class RenderManifest
{
public string AepPath { get; set; } = "";
public string CompName { get; set; } = "";
public string OutputDir { get; set; } = "";
public string OutputPattern { get; set; } = "";
public string? OutputModuleTemplate { get; set; }
public string? RenderSettingsTemplate { get; set; }
public int FrameStart { get; set; }
public int FrameEnd { get; set; }
[JsonExtensionData]
public Dictionary<string, JsonElement>? Extra { get; set; }
public int TotalFrames => FrameEnd - FrameStart + 1;
}
public sealed class ProgressResponse
{
public bool Ok { get; set; }
public bool CancelRequested { get; set; }
public DateTimeOffset? LeaseExpiresAt { get; set; }
}
public sealed class HeartbeatResponse
{
public bool Ok { get; set; }
public List<WorkerCommand> Commands { get; set; } = new();
}
public sealed class WorkerCommand
{
public string Type { get; set; } = "";
public string? JobId { get; set; }
}
public sealed class PresignResponse
{
public string PresignedUrl { get; set; } = "";
public string Key { get; set; } = "";
public string Url { get; set; } = "";
}
public sealed class ExportDetailResponse
{
public ExportDetail Export { get; set; } = new();
}
public sealed class ExportDetail
{
public string Id { get; set; } = "";
public string Status { get; set; } = "";
public List<RenderJobDetail> RenderJobs { get; set; } = new();
}
public sealed class RenderJobDetail
{
public string Id { get; set; } = "";
public string Status { get; set; } = "";
}
/// <summary>Durable lifecycle report persisted to the spool directory (§7.6).</summary>
public sealed class SpooledReport
{
public string Kind { get; set; } = ""; // "fail" | "complete-render"
public string JobId { get; set; } = "";
public Dictionary<string, object?> Payload { get; set; } = new();
public DateTimeOffset CreatedAt { get; set; } = DateTimeOffset.UtcNow;
}
/// <summary>current-job.json — crash recovery state (§7.5).</summary>
public sealed class CurrentJobState
{
public string JobId { get; set; } = "";
public string? ExportId { get; set; } = "";
public string MachineId { get; set; } = "";
public string OutputDirLocal { get; set; } = "";
public DateTimeOffset ClaimedAt { get; set; }
}
@@ -0,0 +1,82 @@
using Microsoft.Extensions.Logging;
namespace VFXReviewWorker;
/// <summary>
/// Minimal rolling file logger (§7.9): one file per day under
/// %ProgramData%\VFXReviewWorker\logs, files older than 14 days pruned on
/// startup and at midnight rollover. No external logging dependencies.
/// </summary>
public sealed class FileLoggerProvider : ILoggerProvider
{
private readonly object _lock = new();
private StreamWriter? _writer;
private DateOnly _currentDay;
public FileLoggerProvider()
{
Directory.CreateDirectory(WorkerOptions.LogsDir);
Prune();
}
public ILogger CreateLogger(string categoryName) => new FileLogger(this, categoryName);
internal void Write(string category, LogLevel level, string message, Exception? ex)
{
lock (_lock)
{
var today = DateOnly.FromDateTime(DateTime.Now);
if (_writer is null || today != _currentDay)
{
_writer?.Dispose();
_currentDay = today;
_writer = new StreamWriter(
Path.Combine(WorkerOptions.LogsDir, $"worker_{today:yyyyMMdd}.log"),
append: true) { AutoFlush = true };
Prune();
}
_writer.WriteLine($"{DateTime.Now:HH:mm:ss.fff} [{level,-5}] {Shorten(category)}: {message}{(ex is null ? "" : $"\n{ex}")}");
}
}
private static string Shorten(string category) => category[(category.LastIndexOf('.') + 1)..];
private static void Prune()
{
try
{
var cutoff = DateTime.Now.AddDays(-14);
foreach (var f in Directory.EnumerateFiles(WorkerOptions.LogsDir, "*.log"))
{
if (File.GetLastWriteTime(f) < cutoff) File.Delete(f);
}
}
catch { /* best effort */ }
}
public void Dispose()
{
lock (_lock) { _writer?.Dispose(); _writer = null; }
}
private sealed class FileLogger : ILogger
{
private readonly FileLoggerProvider _provider;
private readonly string _category;
public FileLogger(FileLoggerProvider provider, string category)
{
_provider = provider;
_category = category;
}
public IDisposable? BeginScope<TState>(TState state) where TState : notnull => null;
public bool IsEnabled(LogLevel logLevel) => logLevel >= LogLevel.Information;
public void Log<TState>(LogLevel logLevel, EventId eventId, TState state, Exception? exception, Func<TState, Exception?, string> formatter)
{
if (!IsEnabled(logLevel)) return;
_provider.Write(_category, logLevel, formatter(state, exception), exception);
}
}
}
+326
View File
@@ -0,0 +1,326 @@
using System.Text.Json;
using Microsoft.Extensions.Logging;
namespace VFXReviewWorker;
/// <summary>
/// Executes one claimed job end-to-end: writes crash-recovery state, runs
/// aerender, uploads the full log, and enqueues the durable complete/fail
/// report. One job at a time per machine (§7.2).
/// </summary>
public sealed class JobExecutor
{
private readonly WorkerOptions _options;
private readonly ApiClient _api;
private readonly AerenderRunner _runner;
private readonly PreviewStage _preview;
private readonly ReportSpooler _spooler;
private readonly ILogger<JobExecutor> _log;
public JobExecutor(WorkerOptions options, ApiClient api, AerenderRunner runner, PreviewStage preview, ReportSpooler spooler, ILogger<JobExecutor> log)
{
_options = options;
_api = api;
_runner = runner;
_preview = preview;
_spooler = spooler;
_log = log;
}
public async Task RunAsync(ClaimedJob job, string machineId, ServerConfig config, CancellationToken cancelJob, CancellationToken shutdown)
{
if (job.Type == "PREVIEW_ONLY")
{
await RunPreviewAsync(job, machineId, config, cancelJob, shutdown);
return;
}
var manifest = job.AsRenderManifest();
var mapper = new PathMapper(_options.PathMappings);
var outputDirLocal = mapper.Map(manifest.OutputDir);
WriteState(new CurrentJobState
{
JobId = job.Id,
ExportId = job.ExportId,
MachineId = machineId,
OutputDirLocal = outputDirLocal,
ClaimedAt = DateTimeOffset.UtcNow,
});
try
{
var result = await _runner.RunAsync(job, manifest, machineId, outputDirLocal, config, cancelJob, shutdown);
var logKey = await _api.UploadLogAsync(job.Id, machineId, result.FullLogPath, shutdown);
if (result.Success)
{
long totalBytes = 0;
int fileCount = 0;
var prefix = AerenderRunner.OutputPatternPrefix(manifest.OutputPattern);
try
{
foreach (var f in Directory.EnumerateFiles(outputDirLocal))
{
if (!Path.GetFileName(f).StartsWith(prefix, StringComparison.OrdinalIgnoreCase)) continue;
fileCount++;
totalBytes += new FileInfo(f).Length;
}
}
catch { /* stats are best-effort in Phase 2; validation owns this in Phase 3 */ }
if (fileCount == 0)
{
// aerender can exit 0 having rendered nothing (bad frame
// range, template mismatch, silently swallowed error).
// Until Phase 3 validation exists, an empty output dir must
// never become READY_FOR_QC.
_log.LogError("Job {JobId}: aerender exited 0 but no \"{Prefix}*\" files exist in {Dir} — reporting failure",
job.Id, prefix, outputDirLocal);
_spooler.Enqueue("fail", job.Id, new Dictionary<string, object?>
{
["machineId"] = machineId,
["stage"] = "RENDER",
["exitCode"] = 0,
["errorMessage"] = "aerender exited 0 but produced no output files in " + outputDirLocal,
["logTail"] = result.LogTail,
["logFileKey"] = logKey,
["retryable"] = false,
});
return;
}
_log.LogInformation("Job {JobId} complete: {Frames} frames in {Sec:F0}s ({Files} files, {Bytes} bytes)",
job.Id, manifest.TotalFrames, result.RenderSeconds, fileCount, totalBytes);
_spooler.Enqueue("complete-render", job.Id, new Dictionary<string, object?>
{
["machineId"] = machineId,
["renderSeconds"] = (int)result.RenderSeconds,
["logFileKey"] = logKey,
["logTail"] = result.LogTail,
["exrFileCount"] = fileCount,
["exrTotalBytes"] = totalBytes,
});
}
else if (result.ErrorMessage == "Cancelled")
{
// Ack the cancel so the server clears the CANCEL_JOB command
_log.LogInformation("Job {JobId} cancelled — acknowledging", job.Id);
_spooler.Enqueue("fail", job.Id, new Dictionary<string, object?>
{
["machineId"] = machineId,
["stage"] = "RENDER",
["errorMessage"] = "Cancelled by server",
["logTail"] = result.LogTail,
["logFileKey"] = logKey,
["retryable"] = false,
});
}
else
{
_log.LogError("Job {JobId} failed (exit {Exit}, retryable {Retryable}): {Error}",
job.Id, result.ExitCode, result.Retryable, result.ErrorMessage);
_spooler.Enqueue("fail", job.Id, new Dictionary<string, object?>
{
["machineId"] = machineId,
["stage"] = "RENDER",
["exitCode"] = result.ExitCode,
["errorMessage"] = result.ErrorMessage,
["logTail"] = result.LogTail,
["logFileKey"] = logKey,
["retryable"] = result.Retryable,
});
}
}
catch (Exception ex)
{
_log.LogError(ex, "Unexpected executor error for job {JobId}", job.Id);
_spooler.Enqueue("fail", job.Id, new Dictionary<string, object?>
{
["machineId"] = machineId,
["stage"] = "RENDER",
["errorMessage"] = $"Worker exception: {ex.Message}",
["retryable"] = true,
});
}
finally
{
ClearState();
}
}
/// <summary>
/// PREVIEW_ONLY job: rebuild the shot around the rendered EXRs with slate
/// and burn-ins, render the delivery MOV + review MP4, upload the review
/// media, and finalize (which registers the internal-only Version).
/// </summary>
private async Task RunPreviewAsync(ClaimedJob job, string machineId, ServerConfig config, CancellationToken cancelJob, CancellationToken shutdown)
{
var manifest = job.AsPreviewManifest();
WriteState(new CurrentJobState
{
JobId = job.Id,
ExportId = job.ExportId,
MachineId = machineId,
OutputDirLocal = new PathMapper(_options.PathMappings).Map(manifest.OutputDir),
ClaimedAt = DateTimeOffset.UtcNow,
});
try
{
var result = await _preview.RunAsync(job, manifest, machineId, config, cancelJob, shutdown);
if (!result.Success)
{
_log.LogError("Preview job {JobId} failed (retryable {Retryable}): {Error}",
job.Id, result.Retryable, result.ErrorMessage);
_spooler.Enqueue("fail", job.Id, new Dictionary<string, object?>
{
["machineId"] = machineId,
["stage"] = "PREVIEW",
["errorMessage"] = result.ErrorMessage,
["retryable"] = result.Retryable,
});
return; // temp AEP is deliberately kept for debugging
}
var previewKey = await _api.UploadArtifactAsync(
job.Id, machineId, "preview", result.Mp4Path!, "video/mp4", shutdown);
if (previewKey is null)
{
// The renders are on the SAN and fine — only the upload failed,
// so this is worth retrying without rebuilding anything.
_spooler.Enqueue("fail", job.Id, new Dictionary<string, object?>
{
["machineId"] = machineId,
["stage"] = "PREVIEW",
["errorMessage"] = "Preview MP4 upload failed",
["retryable"] = true,
});
return;
}
string? thumbKey = null;
if (result.ThumbnailPath is not null)
{
thumbKey = await _api.UploadArtifactAsync(
job.Id, machineId, "thumbnail", result.ThumbnailPath, "image/jpeg", shutdown);
}
_spooler.Enqueue("finalize", job.Id, new Dictionary<string, object?>
{
["machineId"] = machineId,
["artifacts"] = new Dictionary<string, object?>
{
["deliveryMovPath"] = manifest.MovOutput,
["previewMovKey"] = previewKey,
["thumbnailKey"] = thumbKey,
},
["renderStats"] = new Dictionary<string, object?>
{
["previewSeconds"] = (int)result.Seconds,
},
["media"] = new Dictionary<string, object?>
{
["width"] = result.Build?.Width,
["height"] = result.Build?.Height,
["frameCount"] = result.Build?.FrameCount,
["fps"] = manifest.Fps,
["fileName"] = Path.GetFileName(result.Mp4Path!),
},
});
// Only clean up once the outputs are safely reported.
CleanScratch(job.Id, result.TempAep);
}
catch (Exception ex)
{
_log.LogError(ex, "Unexpected preview error for job {JobId}", job.Id);
_spooler.Enqueue("fail", job.Id, new Dictionary<string, object?>
{
["machineId"] = machineId,
["stage"] = "PREVIEW",
["errorMessage"] = $"Worker exception: {ex.Message}",
["retryable"] = true,
});
}
finally
{
ClearState();
}
}
private static void CleanScratch(string jobId, string? tempAep)
{
PreviewStage.TryDelete(Path.Combine(WorkerOptions.ScratchDir, $"job_{jobId}_context.json"));
PreviewStage.TryDelete(Path.Combine(WorkerOptions.ScratchDir, $"job_{jobId}_result.json"));
PreviewStage.TryDelete(Path.Combine(WorkerOptions.ScratchDir, $"job_{jobId}_run.jsx"));
PreviewStage.TryDelete(Path.Combine(WorkerOptions.ScratchDir, $"job_{jobId}_thumb.jpg"));
if (tempAep is not null) PreviewStage.TryDelete(tempAep);
}
/// <summary>
/// Crash recovery (§7.5): on service start, if current-job.json names a
/// job, ask the server what became of it. Still ours (CLAIMED/RUNNING) →
/// we crashed mid-render; report a retryable fail so the server requeues
/// (never resume a partial aerender). Reassigned/finished → discard state;
/// the next attempt's PrepareOutputDir clears any orphaned frames.
/// </summary>
public async Task ReconcileAsync(string machineId, CancellationToken ct)
{
var state = ReadState();
if (state is null) return;
_log.LogWarning("Found crash-recovery state for job {JobId} — reconciling", state.JobId);
try
{
var stillOurs = false;
if (!string.IsNullOrEmpty(state.ExportId))
{
var export = await _api.GetExportAsync(state.ExportId!, ct);
var job = export?.RenderJobs.FirstOrDefault(j => j.Id == state.JobId);
stillOurs = job?.Status is "CLAIMED" or "RUNNING";
}
if (stillOurs)
{
_log.LogWarning("Server still shows job {JobId} as ours — reporting crash for requeue", state.JobId);
_spooler.Enqueue("fail", state.JobId, new Dictionary<string, object?>
{
["machineId"] = machineId,
["stage"] = "RENDER",
["errorMessage"] = "Worker restarted mid-render (crash/power loss); never resuming a partial render",
["retryable"] = true,
});
}
}
catch (Exception ex)
{
_log.LogWarning(ex, "Reconcile failed for job {JobId}; discarding local state (reaper covers the lease)", state.JobId);
}
ClearState();
}
private static void WriteState(CurrentJobState state)
{
Directory.CreateDirectory(WorkerOptions.DataDir);
File.WriteAllText(WorkerOptions.StateFilePath, JsonSerializer.Serialize(state, WorkerOptions.JsonOpts));
}
private static CurrentJobState? ReadState()
{
try
{
if (!File.Exists(WorkerOptions.StateFilePath)) return null;
return JsonSerializer.Deserialize<CurrentJobState>(File.ReadAllText(WorkerOptions.StateFilePath), WorkerOptions.JsonOpts);
}
catch
{
return null;
}
}
private static void ClearState()
{
try { File.Delete(WorkerOptions.StateFilePath); } catch { }
}
}
@@ -0,0 +1,36 @@
namespace VFXReviewWorker;
/// <summary>
/// Translates canonical UNC paths from manifests to this machine's local
/// drive mappings (§7.8 pathMappings). Comparison is case-insensitive and
/// slash-direction tolerant; the first matching mapping wins.
/// </summary>
public sealed class PathMapper
{
private readonly List<(string From, string To)> _mappings;
public PathMapper(IEnumerable<PathMapping> mappings)
{
_mappings = mappings
.Where(m => !string.IsNullOrWhiteSpace(m.From))
.Select(m => (Normalize(m.From), m.To))
.ToList();
}
private static string Normalize(string p) => p.Replace('\\', '/');
public string Map(string path)
{
if (string.IsNullOrEmpty(path)) return path;
var normalized = Normalize(path);
foreach (var (from, to) in _mappings)
{
if (normalized.StartsWith(from, StringComparison.OrdinalIgnoreCase))
{
var mapped = to + normalized[from.Length..];
return mapped.Replace('/', Path.DirectorySeparatorChar);
}
}
return path.Replace('/', Path.DirectorySeparatorChar);
}
}
@@ -0,0 +1,283 @@
using System.Diagnostics;
using System.Text;
using System.Text.Json;
using Microsoft.Extensions.Logging;
namespace VFXReviewWorker;
public sealed record PreviewStageResult(
bool Success,
bool Retryable,
string? ErrorMessage,
string? MovPath,
string? Mp4Path,
string? ThumbnailPath,
string? TempAep,
PreviewBuildResult? Build,
double Seconds);
/// <summary>
/// Stage 3 — preview generation (RenderPipeline2 §9), the second half of a
/// one-click export.
///
/// Runs in two steps so the unproven part stays small:
/// 1. AfterFX.com -noui -r → builds the shot around the rendered EXRs using
/// the studio slate/overlay template, queues the MOV + MP4 output modules
/// and saves a throwaway AEP. Does no rendering.
/// 2. aerender -project &lt;that aep&gt; (no -comp) → renders the whole queue in
/// one launch, through the same proven runner as the EXR stage.
/// </summary>
public sealed class PreviewStage
{
private readonly WorkerOptions _options;
private readonly AerenderRunner _runner;
private readonly ILogger<PreviewStage> _log;
public PreviewStage(WorkerOptions options, AerenderRunner runner, ILogger<PreviewStage> log)
{
_options = options;
_runner = runner;
_log = log;
}
public async Task<PreviewStageResult> RunAsync(
ClaimedJob job,
PreviewManifest manifest,
string machineId,
ServerConfig config,
CancellationToken cancelJob,
CancellationToken shutdown)
{
var sw = Stopwatch.StartNew();
var mapper = new PathMapper(_options.PathMappings);
Directory.CreateDirectory(WorkerOptions.ScratchDir);
var contextPath = Path.Combine(WorkerOptions.ScratchDir, $"job_{job.Id}_context.json");
var resultPath = Path.Combine(WorkerOptions.ScratchDir, $"job_{job.Id}_result.json");
var wrapperPath = Path.Combine(WorkerOptions.ScratchDir, $"job_{job.Id}_run.jsx");
var tempAep = Path.Combine(WorkerOptions.ScratchDir, $"job_{job.Id}_preview.aep");
var movLocal = mapper.Map(manifest.MovOutput);
var mp4Local = mapper.Map(manifest.Mp4Output);
// Everything the script needs, in machine-local paths.
var context = new
{
jobId = job.Id,
exportId = manifest.ExportId,
shotCode = manifest.ShotCode,
versionString = manifest.VersionString,
outputDirLocal = mapper.Map(manifest.OutputDir),
templateAep = mapper.Map(manifest.TemplateAep),
templateComp = manifest.TemplateComp,
overlayComp = manifest.OverlayComp,
lutComp = manifest.LutComp,
movTemplate = manifest.MovTemplate,
mp4Template = manifest.Mp4Template,
movOutput = movLocal,
mp4Output = mp4Local,
tempAep,
resultPath,
frameStart = manifest.FrameStart,
fps = manifest.Fps,
slateScopeProp = manifest.SlateScopeProp,
slateSubmissionProp = manifest.SlateSubmissionProp,
slate = manifest.Slate,
};
await File.WriteAllTextAsync(contextPath, JsonSerializer.Serialize(context, WorkerOptions.JsonOpts), shutdown);
var scriptPath = _options.ResolvePreviewScriptPath();
if (!File.Exists(scriptPath))
{
return Fail($"Preview build script not found at {scriptPath}", retryable: false, sw);
}
// ExtendScript cannot read argv — hand the context path over through a
// generated wrapper that evaluates the shared build script.
var wrapper = new StringBuilder()
.AppendLine("// generated by VFXReviewWorker — do not edit")
.AppendLine($"var VFXR_CONTEXT_PATH = {JsonSerializer.Serialize(contextPath)};")
.AppendLine($"$.evalFile({JsonSerializer.Serialize(scriptPath)});")
.ToString();
await File.WriteAllTextAsync(wrapperPath, wrapper, shutdown);
TryDelete(resultPath);
// ── Step 1: headless build ───────────────────────────────────────────
var afterFx = _options.ResolveAfterFxPath();
if (!File.Exists(afterFx))
{
return Fail($"AfterFX.com not found at {afterFx}", retryable: false, sw);
}
// Belt and braces: the claim loop already avoids preview jobs while AE
// is open, but an artist may have launched it in between. Never take
// over a live session — defer instead (retryable).
if (AeSession.InteractiveRunning())
{
return Fail(
"An interactive After Effects session is open on this machine — preview build deferred to avoid taking it over",
retryable: true, sw);
}
_log.LogInformation("Building preview comp for job {JobId} via {AfterFx}", job.Id, afterFx);
var buildExit = await RunAfterFxAsync(afterFx, wrapperPath, config, cancelJob, shutdown);
if (!File.Exists(resultPath))
{
return Fail(
$"Headless AE produced no result file (exit {buildExit}). Check that AfterFX.com can run under this account.",
retryable: true, sw);
}
PreviewBuildResult? build;
try
{
build = JsonSerializer.Deserialize<PreviewBuildResult>(
await File.ReadAllTextAsync(resultPath, shutdown), WorkerOptions.JsonOpts);
}
catch (Exception ex)
{
return Fail($"Could not parse preview build result: {ex.Message}", retryable: false, sw);
}
if (build is null || !build.Ok)
{
// A build failure is deterministic (missing template comp, missing
// output module, bad layer name) — humans fix the template.
return Fail($"Preview build failed: {build?.Error ?? "unknown error"}", retryable: false, sw, build);
}
foreach (var w in build.Warnings)
{
_log.LogWarning("Preview build warning (job {JobId}): {Warning}", job.Id, w);
}
if (string.IsNullOrWhiteSpace(build.TempAep) || !File.Exists(build.TempAep))
{
return Fail("Preview build reported success but the temp project is missing", retryable: false, sw, build);
}
// ── Step 2: render the prepared queue ────────────────────────────────
var render = await _runner.RunProjectAsync(
job.Id, machineId, build.TempAep, build.FrameCount ?? manifest.TotalFrames,
config, cancelJob, shutdown);
if (!render.Success)
{
return new PreviewStageResult(false, render.Retryable, render.ErrorMessage,
null, null, null, build.TempAep, build, sw.Elapsed.TotalSeconds);
}
// aerender can exit 0 without writing — verify both outputs exist.
var missing = new List<string>();
if (!File.Exists(movLocal)) missing.Add(movLocal);
if (!File.Exists(mp4Local)) missing.Add(mp4Local);
if (missing.Count > 0)
{
return new PreviewStageResult(false, false,
"aerender exited 0 but expected preview outputs are missing: " + string.Join(", ", missing),
null, null, null, build.TempAep, build, sw.Elapsed.TotalSeconds);
}
var thumbnail = await TryMakeThumbnailAsync(job.Id, mp4Local, shutdown);
sw.Stop();
_log.LogInformation("Preview complete for job {JobId} in {Sec:F0}s (mov + mp4{Thumb})",
job.Id, sw.Elapsed.TotalSeconds, thumbnail is null ? "" : " + thumbnail");
return new PreviewStageResult(true, true, null, movLocal, mp4Local, thumbnail,
build.TempAep, build, sw.Elapsed.TotalSeconds);
}
private async Task<int> RunAfterFxAsync(
string afterFx, string wrapperPath, ServerConfig config,
CancellationToken cancelJob, CancellationToken shutdown)
{
var psi = new ProcessStartInfo
{
FileName = afterFx,
RedirectStandardOutput = true,
RedirectStandardError = true,
UseShellExecute = false,
CreateNoWindow = true,
};
psi.ArgumentList.Add("-noui");
psi.ArgumentList.Add("-r");
psi.ArgumentList.Add(wrapperPath);
using var process = new Process { StartInfo = psi };
process.OutputDataReceived += (_, e) => { if (!string.IsNullOrWhiteSpace(e.Data)) _log.LogDebug("[AfterFX] {Line}", e.Data); };
process.ErrorDataReceived += (_, e) => { if (!string.IsNullOrWhiteSpace(e.Data)) _log.LogDebug("[AfterFX] {Line}", e.Data); };
process.Start();
process.BeginOutputReadLine();
process.BeginErrorReadLine();
// The build does no rendering, so it should finish in well under the
// stall timeout. A hang here means AE is waiting on something (a dialog,
// a licence prompt) and must be killed rather than left holding the job.
using var timeout = CancellationTokenSource.CreateLinkedTokenSource(cancelJob, shutdown);
timeout.CancelAfter(TimeSpan.FromSeconds(Math.Max(120, config.StallTimeoutSeconds)));
try
{
await process.WaitForExitAsync(timeout.Token);
}
catch (OperationCanceledException)
{
_log.LogError("Headless AE build timed out or was cancelled — killing process tree");
try { process.Kill(entireProcessTree: true); } catch { }
return -1;
}
return process.ExitCode;
}
/// <summary>Optional 960-wide poster frame at 25% duration (§9.1). Never fatal.</summary>
private async Task<string?> TryMakeThumbnailAsync(string jobId, string mp4Path, CancellationToken ct)
{
if (string.IsNullOrWhiteSpace(_options.FfmpegPath) || !File.Exists(_options.FfmpegPath))
{
return null;
}
var outPath = Path.Combine(WorkerOptions.ScratchDir, $"job_{jobId}_thumb.jpg");
TryDelete(outPath);
var psi = new ProcessStartInfo
{
FileName = _options.FfmpegPath,
RedirectStandardOutput = true,
RedirectStandardError = true,
UseShellExecute = false,
CreateNoWindow = true,
};
foreach (var a in new[] { "-y", "-i", mp4Path, "-vf", "thumbnail,scale=960:-1", "-frames:v", "1", outPath })
{
psi.ArgumentList.Add(a);
}
try
{
using var process = Process.Start(psi);
if (process is null) return null;
using var timeout = CancellationTokenSource.CreateLinkedTokenSource(ct);
timeout.CancelAfter(TimeSpan.FromMinutes(2));
await process.WaitForExitAsync(timeout.Token);
return File.Exists(outPath) ? outPath : null;
}
catch (Exception ex)
{
_log.LogWarning(ex, "Thumbnail generation failed (non-fatal)");
return null;
}
}
private static PreviewStageResult Fail(string message, bool retryable, Stopwatch sw, PreviewBuildResult? build = null)
{
sw.Stop();
return new PreviewStageResult(false, retryable, message, null, null, null, build?.TempAep, build, sw.Elapsed.TotalSeconds);
}
internal static void TryDelete(string path)
{
try { if (File.Exists(path)) File.Delete(path); } catch { }
}
}
+30
View File
@@ -0,0 +1,30 @@
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Logging;
using VFXReviewWorker;
// VFXReview RenderWorker (RenderPipeline2 §7) — Windows service driving
// aerender.exe on artist workstations / render nodes. All state flows through
// /api/ext/* with API-key auth; the worker never touches the database.
//
// Run interactively for debugging: VFXReviewWorker.exe [path\to\config.json]
// Install as a service: see RenderWorker/README.md
var configPath = args.FirstOrDefault(a => !a.StartsWith('-'));
var options = WorkerOptions.Load(configPath);
// Positional args (config path) are consumed above and deliberately not
// forwarded — the command-line configuration provider rejects bare tokens.
var builder = Host.CreateApplicationBuilder();
builder.Services.AddWindowsService(o => o.ServiceName = "VFXReviewRenderWorker");
builder.Logging.AddProvider(new FileLoggerProvider());
builder.Services.AddSingleton(options);
builder.Services.AddSingleton<ApiClient>();
builder.Services.AddSingleton<ReportSpooler>();
builder.Services.AddSingleton<AerenderRunner>();
builder.Services.AddSingleton<PreviewStage>();
builder.Services.AddSingleton<JobExecutor>();
builder.Services.AddHostedService<WorkerService>();
await builder.Build().RunAsync();
@@ -0,0 +1,80 @@
using System.Text.RegularExpressions;
namespace VFXReviewWorker;
/// <summary>
/// Parses aerender stdout line-by-line (§7.3): PROGRESS lines yield the
/// current frame; ERROR lines are collected; deterministic errors (missing
/// footage/comp/project) mark the job non-retryable — humans fix the comp.
/// </summary>
public static partial class ProgressParser
{
// e.g. "PROGRESS: 0:00:02:03 (51): 0 Seconds" / "PROGRESS: 2 (2): ..."
[GeneratedRegex(@"^PROGRESS:.*\((\d+)\)", RegexOptions.Compiled)]
private static partial Regex ProgressLine();
[GeneratedRegex(@"aerender\s+(SYNTAX\s+)?ERROR|^ERROR:", RegexOptions.Compiled | RegexOptions.IgnoreCase)]
private static partial Regex ErrorLine();
private static readonly string[] NonRetryablePatterns =
{
"missing footage",
"layer source file missing",
"file is missing",
"no comp was found",
"no composition",
"project file could not be opened",
"can not be opened",
"cannot be opened",
"the file format module could not parse the file",
"no render settings template",
"no output module template",
"syntax error",
"illegal argument",
};
/// <summary>Returns the rendered-frame ordinal (1-based within the range) or null.</summary>
public static int? ParseFrame(string line)
{
var m = ProgressLine().Match(line);
return m.Success && int.TryParse(m.Groups[1].Value, out var f) ? f : null;
}
public static bool IsErrorLine(string line) => ErrorLine().IsMatch(line);
/// <summary>Deterministic failures re-render identically — don't burn retries on them.</summary>
public static bool IsNonRetryableError(string line)
{
foreach (var p in NonRetryablePatterns)
{
if (line.Contains(p, StringComparison.OrdinalIgnoreCase)) return true;
}
return false;
}
}
/// <summary>ETA = rolling average seconds/frame × frames remaining (§7.3).</summary>
public sealed class EtaCalculator
{
private readonly int _windowSize;
private readonly Queue<double> _frameSeconds = new();
private DateTimeOffset? _lastFrameAt;
public EtaCalculator(int windowSize = 10) => _windowSize = windowSize;
public void RecordFrame(DateTimeOffset now)
{
if (_lastFrameAt is { } last)
{
_frameSeconds.Enqueue((now - last).TotalSeconds);
while (_frameSeconds.Count > _windowSize) _frameSeconds.Dequeue();
}
_lastFrameAt = now;
}
public int? EstimateSeconds(int framesRemaining)
{
if (_frameSeconds.Count == 0 || framesRemaining <= 0) return null;
return (int)Math.Round(_frameSeconds.Average() * framesRemaining);
}
}
@@ -0,0 +1,91 @@
using System.Text.Json;
using Microsoft.Extensions.Logging;
namespace VFXReviewWorker;
/// <summary>
/// Durable lifecycle reports (§7.6): fail/complete reports are written to the
/// spool directory before the first send attempt and replayed strictly in
/// order with exponential backoff (1 s → 60 s). Rendering continues during
/// server downtime; a report is deleted only once the server accepts it (or
/// permanently rejects it as contradictory).
/// </summary>
public sealed class ReportSpooler
{
private readonly ApiClient _api;
private readonly ILogger<ReportSpooler> _log;
private readonly string _dir;
private readonly SemaphoreSlim _wake = new(0);
private long _seq;
public ReportSpooler(ApiClient api, ILogger<ReportSpooler> log, string? dir = null)
{
_api = api;
_log = log;
_dir = dir ?? WorkerOptions.SpoolDir;
Directory.CreateDirectory(_dir);
}
public void Enqueue(string kind, string jobId, Dictionary<string, object?> payload)
{
var report = new SpooledReport { Kind = kind, JobId = jobId, Payload = payload };
var name = $"{DateTimeOffset.UtcNow.UtcTicks:D20}_{Interlocked.Increment(ref _seq):D6}.json";
var tmp = Path.Combine(_dir, name + ".tmp");
var final = Path.Combine(_dir, name);
File.WriteAllText(tmp, JsonSerializer.Serialize(report, WorkerOptions.JsonOpts));
File.Move(tmp, final);
_wake.Release();
}
public bool HasPending => Directory.EnumerateFiles(_dir, "*.json").Any();
/// <summary>Long-running replay loop; started once by the worker service.</summary>
public async Task RunAsync(CancellationToken ct)
{
var backoff = TimeSpan.FromSeconds(1);
while (!ct.IsCancellationRequested)
{
var next = Directory.EnumerateFiles(_dir, "*.json").OrderBy(f => f, StringComparer.Ordinal).FirstOrDefault();
if (next is null)
{
try { await _wake.WaitAsync(TimeSpan.FromSeconds(5), ct); } catch (OperationCanceledException) { break; }
continue;
}
bool done;
try
{
var report = JsonSerializer.Deserialize<SpooledReport>(File.ReadAllText(next), WorkerOptions.JsonOpts);
if (report is null || string.IsNullOrEmpty(report.JobId))
{
_log.LogWarning("Discarding unreadable spool file {File}", next);
done = true;
}
else
{
done = await _api.SendLifecycleAsync(report.Kind, report.JobId, report.Payload, ct);
}
}
catch (OperationCanceledException)
{
break;
}
catch (Exception ex)
{
_log.LogWarning(ex, "Spool replay attempt failed for {File}", next);
done = false;
}
if (done)
{
try { File.Delete(next); } catch { /* re-sent next pass; server side is idempotent */ }
backoff = TimeSpan.FromSeconds(1);
}
else
{
try { await Task.Delay(backoff, ct); } catch (OperationCanceledException) { break; }
backoff = TimeSpan.FromSeconds(Math.Min(backoff.TotalSeconds * 2, 60));
}
}
}
}
@@ -0,0 +1,29 @@
<Project Sdk="Microsoft.NET.Sdk.Worker">
<PropertyGroup>
<TargetFramework>net8.0</TargetFramework>
<Nullable>enable</Nullable>
<ImplicitUsings>enable</ImplicitUsings>
<RootNamespace>VFXReviewWorker</RootNamespace>
<AssemblyName>VFXReviewWorker</AssemblyName>
<!-- Single self-contained exe for render nodes: dotnet publish -r win-x64 (see README) -->
<InvariantGlobalization>true</InvariantGlobalization>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="Microsoft.Extensions.Hosting" Version="8.0.1" />
<PackageReference Include="Microsoft.Extensions.Hosting.WindowsServices" Version="8.0.1" />
</ItemGroup>
<ItemGroup>
<InternalsVisibleTo Include="VFXReviewWorker.Tests" />
</ItemGroup>
<ItemGroup>
<!-- Headless AE build script ships beside the exe -->
<Content Include="..\scripts\vfxr_build_preview.jsx" Link="scripts\vfxr_build_preview.jsx">
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
</Content>
</ItemGroup>
</Project>
@@ -0,0 +1,79 @@
using System.Text.Json;
using System.Text.Json.Serialization;
namespace VFXReviewWorker;
/// <summary>
/// Machine-local config (RenderPipeline2 §7.8) — deliberately minimal;
/// everything tunable lives in server SystemConfig and arrives via E6.
/// Default path: C:\ProgramData\VFXReviewWorker\config.json
/// </summary>
public sealed class WorkerOptions
{
public string ServerUrl { get; set; } = "";
public string ApiKey { get; set; } = "";
public string MachineName { get; set; } = Environment.MachineName;
public string AerenderPath { get; set; } = @"C:\Program Files\Adobe\Adobe After Effects 2026\Support Files\aerender.exe";
/// <summary>AfterFX.com — the console AE used for the headless preview build. Defaults beside aerender.</summary>
public string? AfterFxPath { get; set; }
/// <summary>vfxr_build_preview.jsx — defaults to scripts\ beside the worker exe.</summary>
public string? PreviewScriptPath { get; set; }
public string? FfmpegPath { get; set; }
public string? OiiotoolPath { get; set; }
public List<PathMapping> PathMappings { get; set; } = new();
public string WorkerVersion { get; set; } = "1.0.0";
public string? AeVersion { get; set; }
public static string DataDir =>
Path.Combine(Environment.GetFolderPath(Environment.SpecialFolder.CommonApplicationData), "VFXReviewWorker");
public static string DefaultConfigPath => Path.Combine(DataDir, "config.json");
public static string LogsDir => Path.Combine(DataDir, "logs");
public static string SpoolDir => Path.Combine(DataDir, "spool");
public static string ScratchDir => Path.Combine(DataDir, "scratch");
public static string StateFilePath => Path.Combine(DataDir, "current-job.json");
/// <summary>AfterFX.com sits beside aerender.exe in a standard AE install.</summary>
public string ResolveAfterFxPath()
{
if (!string.IsNullOrWhiteSpace(AfterFxPath)) return AfterFxPath;
var dir = Path.GetDirectoryName(AerenderPath);
return dir is null ? "AfterFX.com" : Path.Combine(dir, "AfterFX.com");
}
public string ResolvePreviewScriptPath()
{
if (!string.IsNullOrWhiteSpace(PreviewScriptPath)) return PreviewScriptPath;
return Path.Combine(AppContext.BaseDirectory, "scripts", "vfxr_build_preview.jsx");
}
public static WorkerOptions Load(string? path = null)
{
path ??= DefaultConfigPath;
if (!File.Exists(path))
{
throw new FileNotFoundException(
$"Worker config not found at {path}. Create it with serverUrl, apiKey and machineName (see RenderWorker/README.md).");
}
var json = File.ReadAllText(path);
var opts = JsonSerializer.Deserialize<WorkerOptions>(json, JsonOpts)
?? throw new InvalidOperationException($"Could not parse {path}");
if (string.IsNullOrWhiteSpace(opts.ServerUrl)) throw new InvalidOperationException("config.json: serverUrl is required");
if (string.IsNullOrWhiteSpace(opts.ApiKey)) throw new InvalidOperationException("config.json: apiKey is required");
if (string.IsNullOrWhiteSpace(opts.MachineName)) opts.MachineName = Environment.MachineName;
return opts;
}
public static readonly JsonSerializerOptions JsonOpts = new()
{
PropertyNameCaseInsensitive = true,
PropertyNamingPolicy = JsonNamingPolicy.CamelCase,
DefaultIgnoreCondition = JsonIgnoreCondition.WhenWritingNull,
};
}
public sealed class PathMapping
{
public string From { get; set; } = "";
public string To { get; set; } = "";
}
@@ -0,0 +1,159 @@
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Logging;
namespace VFXReviewWorker;
/// <summary>
/// Main loop (§7.2): register on startup, heartbeat on its own timer (the
/// cancel-signal channel), claim-poll only when idle, one job at a time.
/// Workers poll dumbly — all scheduling policy (render windows, Render Now,
/// urgent priority) lives server-side in the claim endpoint (§7.10).
/// </summary>
public sealed class WorkerService : BackgroundService
{
private readonly WorkerOptions _options;
private readonly ApiClient _api;
private readonly JobExecutor _executor;
private readonly ReportSpooler _spooler;
private readonly ILogger<WorkerService> _log;
private string? _machineId;
private ServerConfig _config = new();
private volatile string? _currentJobId;
private CancellationTokenSource? _cancelCurrentJob;
private bool? _lastAeOpen;
public WorkerService(WorkerOptions options, ApiClient api, JobExecutor executor, ReportSpooler spooler, ILogger<WorkerService> log)
{
_options = options;
_api = api;
_executor = executor;
_spooler = spooler;
_log = log;
}
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
_log.LogInformation("VFXReview RenderWorker {Version} starting as \"{Name}\" → {Server}",
_options.WorkerVersion, _options.MachineName, _options.ServerUrl);
_ = Task.Run(() => _spooler.RunAsync(stoppingToken), stoppingToken);
await RegisterWithRetryAsync(stoppingToken);
await _executor.ReconcileAsync(_machineId!, stoppingToken);
_ = Task.Run(() => HeartbeatLoopAsync(stoppingToken), stoppingToken);
// Claim loop — only while idle
while (!stoppingToken.IsCancellationRequested)
{
try
{
// Preview builds drive a full AE instance, which would hijack an
// artist's open session (§9.1). Simply don't ask for those jobs
// while AE is running — they stay queued for a quiet moment
// instead of being claimed and failed.
var aeOpen = AeSession.InteractiveRunning();
if (aeOpen != _lastAeOpen)
{
_log.LogInformation(aeOpen
? "Interactive After Effects detected — not claiming preview jobs until it closes"
: "No interactive After Effects — preview jobs enabled");
_lastAeOpen = aeOpen;
}
var types = aeOpen
? new[] { "AE_RENDER" }
: new[] { "AE_RENDER", "PREVIEW_ONLY" };
var job = await _api.ClaimAsync(_machineId!, types, stoppingToken);
if (job is not null)
{
_log.LogInformation("Claimed job {JobId} (attempt {Attempt}/{Max}, export {ExportId})",
job.Id, job.Attempt, job.MaxAttempts, job.ExportId);
_cancelCurrentJob = CancellationTokenSource.CreateLinkedTokenSource(CancellationToken.None);
_currentJobId = job.Id;
try
{
await _executor.RunAsync(job, _machineId!, _config, _cancelCurrentJob.Token, stoppingToken);
}
finally
{
_currentJobId = null;
_cancelCurrentJob.Dispose();
_cancelCurrentJob = null;
}
continue; // immediately look for the next job
}
}
catch (OperationCanceledException) when (stoppingToken.IsCancellationRequested)
{
break;
}
catch (Exception ex)
{
_log.LogWarning(ex, "Claim poll failed (server unreachable?) — retrying on normal cadence");
}
try { await Task.Delay(TimeSpan.FromSeconds(_config.PollSeconds), stoppingToken); }
catch (OperationCanceledException) { break; }
}
_log.LogInformation("RenderWorker stopping");
}
private async Task RegisterWithRetryAsync(CancellationToken ct)
{
var backoff = TimeSpan.FromSeconds(1);
while (!ct.IsCancellationRequested)
{
try
{
var reg = await _api.RegisterAsync(_options, ct);
_machineId = reg.Machine.Id;
_config = reg.Config;
_log.LogInformation(
"Registered as machine {Id} ({Name}); poll={Poll}s heartbeat={Hb}s lease={Lease}s stall={Stall}s enabled={Enabled}",
reg.Machine.Id, reg.Machine.Name, _config.PollSeconds, _config.HeartbeatSeconds,
_config.LeaseSeconds, _config.StallTimeoutSeconds, reg.Machine.Enabled);
return;
}
catch (OperationCanceledException) when (ct.IsCancellationRequested)
{
return;
}
catch (Exception ex)
{
_log.LogWarning("Registration failed ({Error}); retrying in {Backoff}s", ex.Message, backoff.TotalSeconds);
try { await Task.Delay(backoff, ct); } catch (OperationCanceledException) { return; }
backoff = TimeSpan.FromSeconds(Math.Min(backoff.TotalSeconds * 2, 60));
}
}
}
private async Task HeartbeatLoopAsync(CancellationToken ct)
{
while (!ct.IsCancellationRequested)
{
try
{
var resp = await _api.HeartbeatAsync(_machineId!, _currentJobId, ct);
foreach (var cmd in resp.Commands)
{
if (cmd.Type == "CANCEL_JOB" && cmd.JobId is not null && cmd.JobId == _currentJobId)
{
_log.LogWarning("Heartbeat carried CANCEL_JOB for current job {JobId}", cmd.JobId);
_cancelCurrentJob?.Cancel();
}
}
}
catch (OperationCanceledException) when (ct.IsCancellationRequested)
{
return;
}
catch (Exception ex)
{
_log.LogDebug(ex, "Heartbeat failed (non-fatal)");
}
try { await Task.Delay(TimeSpan.FromSeconds(_config.HeartbeatSeconds), ct); }
catch (OperationCanceledException) { return; }
}
}
}
+138
View File
@@ -0,0 +1,138 @@
<#
.SYNOPSIS
Headless AE spike / debug tool for the preview build stage.
.DESCRIPTION
Runs vfxr_build_preview.jsx exactly the way the RenderWorker does, but
standalone, so the risky part (AfterFX.com -noui -r under a service
account) can be proven before wiring the whole pipeline together.
It does NOT render - it builds the preview comp, queues the MOV + MP4
output modules and saves a temp AEP, then prints the build result.
IMPORTANT: close After Effects first. `AfterFX.com -r` hands the script to
an already-running instance, which would open the template in your session
and then quit it.
.EXAMPLE
.\test-preview-build.ps1 -ExrDir "V:\_EXPORTS\UNG\renders\UNG_S1\111\UNG_111_001_030\v002" `
-ShotCode "UNG_111_001_030" -Version "v002"
.EXAMPLE
# then render what it built, exactly as the worker would:
& "C:\Program Files\Adobe\Adobe After Effects 2026\Support Files\aerender.exe" -project "$env:ProgramData\VFXReviewWorker\scratch\spike_preview.aep"
#>
[CmdletBinding()]
param(
[Parameter(Mandatory = $true)][string]$ExrDir,
[Parameter(Mandatory = $true)][string]$ShotCode,
[string]$Version = "v001",
[string]$TemplateAep = "X:/shared_projects_2026/UNGO_VFX/production/working_files/UNG_VFXOVERLAY_SLATE_TEMPLATE.aep",
[string]$TemplateComp = "UNG_EXPORT_TEMPLATE",
[string]$OverlayComp = "UNG_VFX_OVERLAY",
[string]$LutComp = "_SHOW LUT",
[string]$MovTemplate = "4444 Tri",
[string]$Mp4Template = "REVIEW_PREVIEW",
[string]$AfterFx = "C:\Program Files\Adobe\Adobe After Effects 2026\Support Files\AfterFX.com",
[int]$FrameStart = 1001,
[double]$Fps = 24,
[int]$TimeoutSeconds = 300
)
$ErrorActionPreference = "Stop"
$running = Get-Process AfterFX -ErrorAction SilentlyContinue
if ($running) {
Write-Warning "After Effects is running (PID $($running.Id -join ', '))."
Write-Warning "AfterFX.com -r would target that instance and quit it. Close AE first."
return
}
$scratch = Join-Path $env:ProgramData "VFXReviewWorker\scratch"
if (-not (Test-Path $scratch)) { New-Item -ItemType Directory -Force -Path $scratch | Out-Null }
$scriptPath = Join-Path $PSScriptRoot "vfxr_build_preview.jsx"
if (-not (Test-Path $scriptPath)) { throw "Build script not found: $scriptPath" }
if (-not (Test-Path $AfterFx)) { throw "AfterFX.com not found: $AfterFx" }
if (-not (Test-Path $ExrDir)) { throw "EXR folder not found: $ExrDir" }
$contextPath = Join-Path $scratch "spike_context.json"
$resultPath = Join-Path $scratch "spike_result.json"
$wrapperPath = Join-Path $scratch "spike_run.jsx"
$tempAep = Join-Path $scratch "spike_preview.aep"
$base = "${ShotCode}_cmp_TT_${Version}"
Remove-Item $resultPath -ErrorAction SilentlyContinue
$context = [ordered]@{
jobId = "spike"
exportId = "spike"
shotCode = $ShotCode
versionString = $Version
outputDirLocal = $ExrDir
templateAep = $TemplateAep
templateComp = $TemplateComp
overlayComp = $OverlayComp
lutComp = $LutComp
movTemplate = $MovTemplate
mp4Template = $Mp4Template
movOutput = (Join-Path $ExrDir "$base.mov")
mp4Output = (Join-Path $ExrDir "$base.mp4")
tempAep = $tempAep
resultPath = $resultPath
frameStart = $FrameStart
fps = $Fps
slate = [ordered]@{
versionName = $base
date = (Get-Date -Format "yyyy/MM/dd")
description = "Spike test"
notes = ""
shotCode = $ShotCode
episode = ""
scene = ""
}
}
$context | ConvertTo-Json -Depth 6 | Out-File -FilePath $contextPath -Encoding utf8
$ctxJs = ($contextPath -replace '\\', '/') | ConvertTo-Json
$scriptJs = ($scriptPath -replace '\\', '/') | ConvertTo-Json
$wrapper = @(
"// generated by test-preview-build.ps1",
("var VFXR_CONTEXT_PATH = {0};" -f $ctxJs),
('$.evalFile({0});' -f $scriptJs)
) -join "`r`n"
$wrapper | Out-File -FilePath $wrapperPath -Encoding utf8
Write-Host "Launching headless AE build..." -ForegroundColor Cyan
$proc = Start-Process -FilePath $AfterFx -ArgumentList @("-noui", "-r", $wrapperPath) -PassThru -NoNewWindow
if (-not $proc.WaitForExit($TimeoutSeconds * 1000)) {
Write-Warning "Timed out after $TimeoutSeconds s - killing AE."
try { $proc.Kill($true) } catch {}
}
if (-not (Test-Path $resultPath)) {
Write-Host "FAILED: no result file produced." -ForegroundColor Red
Write-Host "AE could not run the script headlessly under this account. See RenderWorker/README.md fallbacks." -ForegroundColor Red
return
}
$result = Get-Content $resultPath -Raw | ConvertFrom-Json
if ($result.ok) {
Write-Host "BUILD OK" -ForegroundColor Green
Write-Host " temp AEP : $($result.tempAep)"
Write-Host " preview comp: $($result.previewComp)"
Write-Host (" frames : {0} ({1} x {2})" -f $result.frameCount, $result.width, $result.height)
Write-Host (" queued : {0} output(s)" -f @($result.queued).Count)
foreach ($q in @($result.queued)) {
Write-Host (" - {0}: {1} -> {2}" -f $q.label, $q.template, $q.output)
}
Write-Host ""
Write-Host "Render it with:" -ForegroundColor Cyan
Write-Host (' "{0}\aerender.exe" -project "{1}"' -f (Split-Path $AfterFx), $result.tempAep)
} else {
Write-Host "BUILD FAILED: $($result.error)" -ForegroundColor Red
}
if ($result.warnings -and $result.warnings.Count -gt 0) {
Write-Host "Warnings:" -ForegroundColor Yellow
$result.warnings | ForEach-Object { Write-Host " - $_" -ForegroundColor Yellow }
}
+527
View File
@@ -0,0 +1,527 @@
/*
vfxr_build_preview.jsx — VFXReview render pipeline, preview build stage.
Runs headlessly: AfterFX.com -noui -r <wrapper>.jsx
where the wrapper sets var VFXR_CONTEXT_PATH = "...json"; then
$.evalFile()s this script.
It does NOT render. It opens the studio slate/overlay template, rebuilds
the shot around the *rendered* EXR sequence (OCIO → show LUT → overlay),
duplicates the export template into a preview comp with the slate filled
in, queues the MOV and MP4 output modules, and saves a throwaway AEP.
The worker then runs `aerender -project <that aep>` with no -comp, which
renders the whole queue in one launch.
Build logic mirrors the VFXReviewConnector panel functions
buildShotFromCode / updateOverlayForComp / buildPreviewForShot so the
farm and the artist produce identical slates and burn-ins.
Everything is reported back through the result JSON — never a dialog.
*/
(function vfxrBuildPreview() {
var ctx = null;
var result = {
ok: false,
error: null,
warnings: [],
tempAep: null,
previewComp: null,
queued: [],
frameCount: null,
width: null,
height: null
};
var resultPath = null;
// ── tiny helpers (ES3 / ExtendScript safe) ───────────────────────────────
function readTextFile(path) {
var f = new File(path);
var text;
if (!f.exists) {
return null;
}
f.encoding = "UTF-8";
if (!f.open("r")) {
return null;
}
text = f.read();
f.close();
return text;
}
function writeTextFile(path, text) {
var f = new File(path);
try {
f.encoding = "UTF-8";
if (!f.open("w")) {
return false;
}
f.write(text);
f.close();
return true;
} catch (writeError) {
return false;
}
}
function jsonEscape(value) {
var text = String(value);
var out = "";
var i;
var ch;
var code;
for (i = 0; i < text.length; i += 1) {
ch = text.charAt(i);
code = text.charCodeAt(i);
if (ch === "\"") {
out += "\\\"";
} else if (ch === "\\") {
out += "\\\\";
} else if (ch === "\n") {
out += "\\n";
} else if (ch === "\r") {
out += "\\r";
} else if (ch === "\t") {
out += "\\t";
} else if (code < 32) {
out += "\\u" + ("000" + code.toString(16)).slice(-4);
} else {
out += ch;
}
}
return out;
}
function toJSON(value) {
var parts = [];
var i;
var key;
if (value === null || value === undefined) {
return "null";
}
if (typeof value === "number") {
return isFinite(value) ? String(value) : "null";
}
if (typeof value === "boolean") {
return String(value);
}
if (typeof value === "string") {
return "\"" + jsonEscape(value) + "\"";
}
if (value instanceof Array) {
for (i = 0; i < value.length; i += 1) {
parts.push(toJSON(value[i]));
}
return "[" + parts.join(",") + "]";
}
for (key in value) {
if (value.hasOwnProperty(key) && value[key] !== undefined) {
parts.push("\"" + jsonEscape(key) + "\":" + toJSON(value[key]));
}
}
return "{" + parts.join(",") + "}";
}
function warn(message) {
result.warnings.push(String(message));
}
function findComp(name) {
var i;
var item;
for (i = 1; i <= app.project.numItems; i += 1) {
item = app.project.item(i);
if (item instanceof CompItem && item.name === name) {
return item;
}
}
return null;
}
function findFolder(name) {
var i;
var item;
for (i = 1; i <= app.project.numItems; i += 1) {
item = app.project.item(i);
if (item instanceof FolderItem && item.name === name) {
return item;
}
}
return null;
}
// Mirrors the panel's addOCIOToLayer: effect added but disabled, so the
// artist/farm chain matches exactly.
function addOCIOToLayer(layer) {
var ocio;
var outputProp;
try {
ocio = layer.property("Effects").addProperty("OCIO Color Space Transform");
ocio.enabled = false;
} catch (ocioError) {
warn("OCIO effect unavailable: " + ocioError.toString());
return;
}
try {
outputProp = ocio.property("Output Color Space");
} catch (byNameError) {
outputProp = null;
}
if (!outputProp) {
try {
outputProp = ocio.property(2);
} catch (byIndexError) {
outputProp = null;
}
}
if (outputProp) {
try {
outputProp.setValue(93);
} catch (setError) {
warn("Could not set OCIO output colour space: " + setError.toString());
}
}
}
function setEssentialProperty(layer, propName, value) {
if (value === null || value === undefined || value === "") {
return false;
}
try {
layer.property("Essential Properties").property(propName).setValue(value);
return true;
} catch (setError) {
return false;
}
}
function setEssentialPropertyByIndex(layer, index, value) {
try {
layer.property("Essential Properties").property(index).setValue(value);
return true;
} catch (setError) {
return false;
}
}
function listEssentialPropertyNames(layer) {
var names = [];
var group;
var i;
try {
group = layer.property("Essential Properties");
for (i = 1; i <= group.numProperties; i += 1) {
try {
names.push(i + ":" + group.property(i).name);
} catch (oneError) {
}
}
} catch (groupError) {
}
return names;
}
// Per-submission fields (VFX Scope / Submission Note). The property names
// come from config; if one does not exist, report the template's actual
// property names so it can be corrected without another build.
function setSlateField(layer, propName, value, label) {
if (!propName) {
return;
}
if (value === null || value === undefined || value === "") {
return; // nothing to write; leave the template's own default
}
if (!setEssentialProperty(layer, propName, value)) {
warn(label + ": Essential Property \"" + propName + "\" not found or not settable. " +
"Available: " + listEssentialPropertyNames(layer).join(", "));
}
}
function setCompStartFrame(comp, frameNumber) {
try {
comp.displayStartFrame = frameNumber;
return;
} catch (frameError) {
}
try {
comp.displayStartTime = frameNumber / comp.frameRate;
} catch (timeError) {
}
}
function applyOutputModuleTemplate(outputModule, templateName) {
var templates;
var i;
try {
templates = outputModule.templates;
for (i = 0; i < templates.length; i += 1) {
if (templates[i] === templateName) {
outputModule.applyTemplate(templateName);
return true;
}
}
} catch (templateError) {
}
return false;
}
function importExrSequence(dirPath) {
var folder = new Folder(dirPath);
var files;
var importOptions;
var footage;
if (!folder.exists) {
throw new Error("Render output folder not found: " + dirPath);
}
files = folder.getFiles("*.exr");
if (!files || files.length < 1) {
throw new Error("No EXR files found in " + dirPath);
}
files.sort();
importOptions = new ImportOptions(files[0]);
importOptions.sequence = true;
footage = app.project.importFile(importOptions);
try {
footage.mainSource.conformFrameRate = ctx.fps || 24;
} catch (conformError) {
warn("Could not conform frame rate: " + conformError.toString());
}
return footage;
}
function queueOutput(comp, templateName, outputPath, label) {
var item;
var outputModule;
if (!templateName || !outputPath) {
warn("Skipping " + label + ": no template or output path configured");
return false;
}
try {
item = app.project.renderQueue.items.add(comp);
outputModule = item.outputModule(1);
if (!applyOutputModuleTemplate(outputModule, templateName)) {
item.remove();
throw new Error("Output module template not found: \"" + templateName + "\"");
}
outputModule.file = new File(outputPath);
result.queued.push({ label: label, template: templateName, output: outputPath });
return true;
} catch (queueError) {
try {
if (item) {
item.remove();
}
} catch (removeError) {
}
throw queueError;
}
}
// ── main ─────────────────────────────────────────────────────────────────
try {
if (typeof VFXR_CONTEXT_PATH === "undefined" || !VFXR_CONTEXT_PATH) {
throw new Error("VFXR_CONTEXT_PATH was not set by the wrapper script");
}
var contextText = readTextFile(VFXR_CONTEXT_PATH);
if (!contextText) {
throw new Error("Could not read job context: " + VFXR_CONTEXT_PATH);
}
ctx = eval("(" + contextText + ")");
resultPath = ctx.resultPath;
try {
app.beginSuppressDialogs();
} catch (suppressError) {
}
// 1. Open the studio template (slate, overlay, show LUT, export template)
var templateFile = new File(ctx.templateAep);
if (!templateFile.exists) {
throw new Error("Template project not found: " + ctx.templateAep);
}
app.open(templateFile);
var templateComp = findComp(ctx.templateComp);
if (!templateComp) {
throw new Error("Export template comp \"" + ctx.templateComp + "\" not found in template project");
}
// 2. Import the rendered EXR sequence
var footage = importExrSequence(ctx.outputDirLocal);
var footageFolder = findFolder("_FOOTAGE_4K");
if (footageFolder) {
footage.parentFolder = footageFolder;
}
footage.name = ctx.shotCode + "_" + ctx.versionString;
var width = footage.width;
var height = footage.height;
var duration = footage.duration;
var fps = ctx.fps || 24;
result.width = width;
result.height = height;
result.frameCount = Math.round(duration * fps);
// 3. Footage precomp (mirrors buildShotFromCode)
var precompFolder = findFolder("_PRECOMPS");
var footageComp = app.project.items.addComp(
ctx.shotCode + "_FOOTAGE", width, height, 1, duration, fps);
if (precompFolder) {
footageComp.parentFolder = precompFolder;
}
if (ctx.frameStart !== null && ctx.frameStart !== undefined) {
setCompStartFrame(footageComp, ctx.frameStart);
}
addOCIOToLayer(footageComp.layers.add(footage));
// 4. Shot comp: footage + show LUT + overlay
var shotComp = app.project.items.addComp(ctx.shotCode, width, height, 1, duration, fps);
if (ctx.frameStart !== null && ctx.frameStart !== undefined) {
setCompStartFrame(shotComp, ctx.frameStart);
}
shotComp.layers.add(footageComp);
if (ctx.lutComp) {
var showLut = findComp(ctx.lutComp);
if (showLut) {
var lutLayer = shotComp.layers.add(showLut);
lutLayer.collapseTransformation = true;
} else {
warn("Show LUT comp \"" + ctx.lutComp + "\" not found — rendering without it");
}
}
if (ctx.overlayComp) {
var overlayComp = findComp(ctx.overlayComp);
if (overlayComp) {
var overlayLayer = shotComp.layers.add(overlayComp);
overlayLayer.moveToBeginning();
overlayLayer.enabled = true;
setEssentialProperty(overlayLayer, "DATE", ctx.slate.date);
setEssentialProperty(overlayLayer, "SHOT NAME", ctx.slate.versionName);
} else {
warn("Overlay comp \"" + ctx.overlayComp + "\" not found — rendering without burn-ins");
}
}
// 5. Preview comp from the export template (mirrors buildPreviewForShot)
var previewsFolder = findFolder("_PREVIEWS");
var previewComp = templateComp.duplicate();
previewComp.name = ctx.shotCode + "_PREVIEW";
if (previewsFolder) {
previewComp.parentFolder = previewsFolder;
}
previewComp.layer("SHOT").replaceSource(shotComp, false);
try {
previewComp.layer("THUMBNAIL").replaceSource(shotComp, false);
} catch (thumbError) {
warn("THUMBNAIL layer not updated: " + thumbError.toString());
}
var slateLayer = previewComp.layer("NETFLIX_SLATE");
if (slateLayer) {
// Index-addressed properties match the panel exactly (3 = version
// name, 10 = shot code); the rest are addressed by name.
setEssentialPropertyByIndex(slateLayer, 3, ctx.slate.versionName);
setEssentialProperty(slateLayer, "Date", ctx.slate.date);
setEssentialProperty(slateLayer, "Desc", ctx.slate.description);
setEssentialPropertyByIndex(slateLayer, 10, ctx.slate.shotCode);
setEssentialProperty(slateLayer, "Episode", ctx.slate.episode);
setEssentialProperty(slateLayer, "Scene", ctx.slate.scene);
setEssentialProperty(slateLayer, "Frames", result.frameCount);
setSlateField(slateLayer, ctx.slateScopeProp, ctx.slate.vfxScope, "VFX Scope");
// The slate's Notes field carries this submission's note; when the
// artist left it blank, fall back to the shot's own notes so the
// slate looks the same as an artist-built preview.
setSlateField(slateLayer, ctx.slateSubmissionProp,
(ctx.slate.submissionNote !== null &&
ctx.slate.submissionNote !== undefined &&
ctx.slate.submissionNote !== "")
? ctx.slate.submissionNote
: ctx.slate.notes,
"Submission Note");
} else {
warn("NETFLIX_SLATE layer not found in template — slate fields not filled");
}
var shotLayer = previewComp.layer("SHOT");
var targetOutPoint = shotLayer.inPoint + shotComp.duration;
if (targetOutPoint > previewComp.duration) {
previewComp.duration = targetOutPoint;
}
shotLayer.outPoint = targetOutPoint;
previewComp.duration = targetOutPoint;
result.previewComp = previewComp.name;
// 6. Queue both outputs against the same preview comp. aerender will
// render the whole queue in one launch.
app.project.renderQueue.showWindow(false);
while (app.project.renderQueue.numItems > 0) {
app.project.renderQueue.item(1).remove();
}
queueOutput(previewComp, ctx.movTemplate, ctx.movOutput, "mov");
queueOutput(previewComp, ctx.mp4Template, ctx.mp4Output, "mp4");
if (result.queued.length < 1) {
throw new Error("Nothing was queued — check output module templates");
}
// 7. Save the throwaway project the worker will hand to aerender
var tempFile = new File(ctx.tempAep);
var tempParent = tempFile.parent;
if (tempParent && !tempParent.exists) {
tempParent.create();
}
app.project.save(tempFile);
result.tempAep = ctx.tempAep;
result.ok = true;
} catch (buildError) {
result.ok = false;
result.error = buildError && buildError.toString ? buildError.toString() : String(buildError);
try {
if (buildError && buildError.line) {
result.error += " (line " + buildError.line + ")";
}
} catch (lineError) {
}
}
try {
app.endSuppressDialogs(false);
} catch (endSuppressError) {
}
if (!resultPath && ctx && ctx.resultPath) {
resultPath = ctx.resultPath;
}
if (resultPath) {
writeTextFile(resultPath, toJSON(result));
}
// -noui leaves the app running otherwise; the worker waits on exit.
try {
app.quit();
} catch (quitError) {
}
}());
@@ -0,0 +1,753 @@
# Technical Architecture Report (Current State)
Date: 2026-07-31
Scope: Current implementation only (no redesign suggestions)
---
## 1. Database
Authoritative schema source: [prisma/schema.prisma](prisma/schema.prisma)
### Full Prisma Schema
The full schema is defined in [prisma/schema.prisma](prisma/schema.prisma).
### Models related to requested domains
- Users/Auth
- User
- Account
- Session
- VerificationToken
- ClientAccess
- Projects/Episodes
- Project
- EpisodeDueDate
- Client
- Shots/Tasks/Versions/Reviews
- Shot
- ShotGroup
- Task
- Version
- Comment
- CommentReply
- Annotation
- Approval
- ReviewSession
- Files/Storage
- FootagePlate
- ShotReference
- SystemConfig
- Delivery/Export adjacent
- Shot fields: highResKey, highResFilename, exrOutput, shotVersion
- Version fields: fileUrl, fileName, proxyUrl, thumbnailUrl, posterUrl
### Relationship summary
- Client 1:N Project
- Project 1:N Shot
- Project 1:N Task
- Project 1:N ReviewSession
- Project 1:N EpisodeDueDate
- Shot 1:N Task
- Shot 1:N Version
- Shot 1:N FootagePlate
- Shot 1:N ShotReference
- Task 1:N Version
- Version 1:N Comment
- Version 1:N Annotation
- Version 1:N Approval
- Comment 1:N CommentReply
- User has many assigned/created entities across shots, tasks, versions, comments, approvals
### Existing status enums
From [prisma/schema.prisma](prisma/schema.prisma):
- ProjectStatus: ACTIVE, ON_HOLD, COMPLETED, ARCHIVED
- ShotStatus: WAITING, IN_PROGRESS, INTERNAL_REVIEW, READY_FOR_CLIENT, CLIENT_REVIEW, REVISIONS, COMPLETE
- ShotApprovalStatus: PENDING, INTERNALLY_APPROVED, CLIENT_APPROVED
- TaskStatus: TODO, IN_PROGRESS, INTERNAL_REVIEW, CLIENT_REVIEW, CHANGES, DONE
- ApprovalStatus: PENDING_REVIEW, APPROVED, REJECTED, NEEDS_CHANGES
- ReviewStatus: PENDING, INTERNAL_APPROVED, CLIENT_APPROVED, NEEDS_CHANGES, FINAL_APPROVED
Notes:
- There is no dedicated Delivery model.
- There is no dedicated Export model.
- Delivery/export state is represented by file pointers and shot/version metadata fields.
---
## 2. API
### 2.1 Shots
#### External shot APIs
- GET /api/ext/projects
- URL: /api/ext/projects
- Method: GET
- Purpose: List projects for pipeline tools
- Request body: None
- Response: projects[] with id, name, code, showId, projectType, status, dates, _count
- Source: [app/api/ext/projects/route.ts](app/api/ext/projects/route.ts)
- GET /api/ext/projects/{projectCode}/episodes
- URL: /api/ext/projects/{projectCode}/episodes
- Method: GET
- Purpose: List distinct episodes and optionally shot payloads per episode
- Request body: None
- Response: project + episodes[]; optional shots[] includes exrOutput/timecodes
- Source: [app/api/ext/projects/[projectCode]/episodes/route.ts](app/api/ext/projects/%5BprojectCode%5D/episodes/route.ts)
- GET /api/ext/projects/{projectCode}/shots
- URL: /api/ext/projects/{projectCode}/shots
- Method: GET
- Purpose: List shots by project code with filters/pagination
- Request body: None
- Response: project + pagination + shots[]
- Source: [app/api/ext/projects/[projectCode]/shots/route.ts](app/api/ext/projects/%5BprojectCode%5D/shots/route.ts)
- GET /api/ext/shots
- URL: /api/ext/shots
- Method: GET
- Purpose: Legacy listing by projectId
- Request body: None
- Response: shots[] + total
- Source: [app/api/ext/shots/route.ts](app/api/ext/shots/route.ts)
- POST /api/ext/shots
- URL: /api/ext/shots
- Method: POST
- Purpose: Create shot from external tool (JSON or multipart thumbnail)
- Request body:
- projectId, scene
- optional episode, description, artistId, priority, fps, frameStart, frameEnd, dueDate, thumbnailUrl, shotGroupName, shotCode
- optional thumbnail file (multipart)
- Response: created shot object
- Source: [app/api/ext/shots/route.ts](app/api/ext/shots/route.ts)
- GET /api/ext/shots/lookup
- URL: /api/ext/shots/lookup
- Method: GET
- Purpose: Canonical shot lookup by shotCode (+ optional projectCode)
- Request body: None
- Response: shot object with project, artist, tasks, latest version, exrOutput, shotVersion, source/seq timecodes
- Source: [app/api/ext/shots/lookup/route.ts](app/api/ext/shots/lookup/route.ts)
- GET /api/ext/shots/{shotId}
- URL: /api/ext/shots/{shotId}
- Method: GET
- Purpose: Shot detail by DB id, or byCode mode
- Request body: None
- Response: full shot detail with tasks/latest version/counts
- Source: [app/api/ext/shots/[shotId]/route.ts](app/api/ext/shots/%5BshotId%5D/route.ts)
- PATCH /api/ext/shots/{shotId}
- URL: /api/ext/shots/{shotId}
- Method: PATCH
- Purpose: Update mutable shot field(s) from pipeline tools
- Request body: shotVersion (v###) currently supported
- Response: success + updated shot id/shotVersion
- Source: [app/api/ext/shots/[shotId]/route.ts](app/api/ext/shots/%5BshotId%5D/route.ts)
#### Internal shot APIs
- GET /api/shots/{shotId}
- URL: /api/shots/{shotId}
- Method: GET
- Purpose: Internal dashboard shot detail payload
- Request body: None
- Response: shot + tasks + artists + permissions flags
- Source: [app/api/shots/[shotId]/route.ts](app/api/shots/%5BshotId%5D/route.ts)
- GET /api/projects/{projectId}/episodes
- URL: /api/projects/{projectId}/episodes
- Method: GET
- Purpose: Internal distinct episode list for project
- Request body: None
- Response: episodes[]
- Source: [app/api/projects/[projectId]/episodes/route.ts](app/api/projects/%5BprojectId%5D/episodes/route.ts)
### 2.2 Reviews
- GET /api/review-sessions
- Purpose: list review sessions
- POST /api/review-sessions
- Purpose: create review session token and portal link
- DELETE /api/review-sessions
- Purpose: deactivate review session
- Source: [app/api/review-sessions/route.ts](app/api/review-sessions/route.ts)
- POST /api/client/{token}/auth
- Purpose: review password check + unlock cookie
- Source: [app/api/client/[token]/auth/route.ts](app/api/client/%5Btoken%5D/auth/route.ts)
- GET /api/client/{token}/project
- Purpose: client portal project payload (shared shots/versions)
- Source: [app/api/client/[token]/project/route.ts](app/api/client/%5Btoken%5D/project/route.ts)
- GET /api/client/{token}/versions/{versionId}
- Purpose: client review detail payload
- Source: [app/api/client/[token]/versions/[versionId]/route.ts](app/api/client/%5Btoken%5D/versions/%5BversionId%5D/route.ts)
- POST /api/client/{token}/comment
- Purpose: client frame comment
- Request body: versionId, frameNumber, timestamp, text
- Response: created comment
- Source: [app/api/client/[token]/comment/route.ts](app/api/client/%5Btoken%5D/comment/route.ts)
- POST /api/client/{token}/annotation
- Purpose: client annotation write
- Request body: versionId, frameNumber, drawingData, optional color
- Response: annotation
- Source: [app/api/client/[token]/annotation/route.ts](app/api/client/%5Btoken%5D/annotation/route.ts)
- POST /api/client/{token}/approve
- Purpose: shot-level approve/changes and legacy version-level approval
- Request body:
- shot mode: shotId + action
- version mode: versionId + status + notes
- Response: success
- Source: [app/api/client/[token]/approve/route.ts](app/api/client/%5Btoken%5D/approve/route.ts)
- GET /api/versions/{versionId}/comments
- Purpose: internal comments read
- Source: [app/api/versions/[versionId]/comments/route.ts](app/api/versions/%5BversionId%5D/comments/route.ts)
- GET /api/versions/{versionId}/annotations
- Purpose: internal annotations read
- Source: [app/api/versions/[versionId]/annotations/route.ts](app/api/versions/%5BversionId%5D/annotations/route.ts)
- GET /api/playlist
- Purpose: latest version per shot playlist
- Source: [app/api/playlist/route.ts](app/api/playlist/route.ts)
### 2.3 File uploads
- POST /api/upload
- Purpose: authenticated upload to Hetzner via app server
- Body: multipart file (+ type)
- Response: url, key
- Source: [app/api/upload/route.ts](app/api/upload/route.ts)
- POST /api/upload/local
- Purpose: authenticated video upload path
- Body: multipart file
- Response: url, key
- Source: [app/api/upload/local/route.ts](app/api/upload/local/route.ts)
- POST /api/upload/presign
- Purpose: direct browser->object-storage upload URL
- Body: fileName, contentType, optional folder
- Response: presignedUrl, key, url
- Source: [app/api/upload/presign/route.ts](app/api/upload/presign/route.ts)
- GET/POST /api/uploadthing
- Purpose: UploadThing route handler passthrough (if configured)
- Source: [app/api/uploadthing/route.ts](app/api/uploadthing/route.ts)
- POST /api/batch-upload/presign
- Purpose: presign high-res upload key
- Body: fileName
- Response: presignedUrl, key
- Source: [app/api/batch-upload/presign/route.ts](app/api/batch-upload/presign/route.ts)
- POST /api/batch-upload/preview
- Purpose: classify upload actions before upload
- Body: projectId, fileNames[]
- Response: items[] with statuses (new-version, rename-and-upload, create-task, update-highres, etc)
- Source: [app/api/batch-upload/preview/route.ts](app/api/batch-upload/preview/route.ts)
- POST /api/batch-upload/upload
- Purpose: commit highres key or create version records
- Body:
- update-highres: action, shotId, projectId, key, fileName
- version upload: action, shotId, projectId, file, task routing fields
- Response: success + action result
- Source: [app/api/batch-upload/upload/route.ts](app/api/batch-upload/upload/route.ts)
### 2.4 EXRs / Rendering / Delivery / Storage / Metadata
- GET /api/files/{...key}
- Purpose: file serving, range requests, local/Hetzner routing
- Source: [app/api/files/[...key]/route.ts](app/api/files/%5B...key%5D/route.ts)
- GET/POST /api/admin/migration
- Purpose: local uploads migration status and per-key migration to Hetzner
- Source: [app/api/admin/migration/route.ts](app/api/admin/migration/route.ts)
- POST/DELETE /api/storage-test
- Purpose: upload test object / delete test object
- Source: [app/api/storage-test/route.ts](app/api/storage-test/route.ts)
- POST/DELETE /api/shots/{shotId}/highres
- Purpose: upload/remove high-res deliverable on shot
- Source: [app/api/shots/[shotId]/highres/route.ts](app/api/shots/%5BshotId%5D/highres/route.ts)
- GET /api/shots/{shotId}/highres/download
- Purpose: internal presigned download URL for high-res
- Source: [app/api/shots/[shotId]/highres/download/route.ts](app/api/shots/%5BshotId%5D/highres/download/route.ts)
- GET /api/client/{token}/shots/{shotId}/highres/download
- Purpose: client token-gated presigned high-res download URL
- Source: [app/api/client/[token]/shots/[shotId]/highres/download/route.ts](app/api/client/%5Btoken%5D/shots/%5BshotId%5D/highres/download/route.ts)
- GET/POST/DELETE /api/shots/{shotId}/references
- Purpose: shot reference image management
- Source: [app/api/shots/[shotId]/references/route.ts](app/api/shots/%5BshotId%5D/references/route.ts)
- GET /api/storyboard/pdf
- Purpose: server-rendered printable storyboard HTML with metadata options
- Source: [app/api/storyboard/pdf/route.ts](app/api/storyboard/pdf/route.ts)
Notes:
- No dedicated REST endpoint that runs ffmpeg in this repository.
- No dedicated REST endpoint that performs EXR rendering on the server.
- No dedicated deliveries API namespace.
---
## 3. Storage
Primary storage abstraction: [lib/storage.ts](lib/storage.ts)
### Provider modes
- local
- uploadthing
- s3
- r2
- b2
- minio
### Dedicated high-res object storage path
- Hetzner object storage helper methods are used for high-res and presigned direct uploads.
- Config source precedence:
- SystemConfig table keys
- env fallback
### Folder/key layout in object storage (current code paths)
- videos/
- image/
- highres/
- storage-test/
### Local storage
- LOCAL_UPLOAD_DIR (default ./uploads)
- Served via /api/files catch-all route
### File naming conventions
- Key format: {folder}/{uuid}-{sanitized-file-name}
- Sanitization done by sanitizeFileName to avoid URL/signature problems in object keys
### EXR locations
Within web app/runtime:
- EXR references are naming metadata (Shot.exrOutput) and file-serving pathing.
Within DCC tooling docs/scripts:
- AE/Nuke workflows target shared export roots and per-shot folders.
### Preview locations
- Version media URLs stored in Version.fileUrl
- Accessed through app player routes and /api/files routing where applicable
### Thumbnail locations
- Shot.thumbnailUrl
- Version.thumbnailUrl
- Can point to /api/files/{key} or external provider URL
Relevant sources:
- [lib/storage.ts](lib/storage.ts)
- [app/api/files/[...key]/route.ts](app/api/files/%5B...key%5D/route.ts)
- [actions/settings.ts](actions/settings.ts)
---
## 4. Export Pipeline (Current)
### AE panel
Documented in repo:
- [VFXReviewConnector.md](VFXReviewConnector.md)
- [EXT_API_REFERENCE.md](EXT_API_REFERENCE.md)
Implemented panel script also exists on the host AE installation (outside workspace), and behavior aligns with docs:
- Episodes/shots discovery via ext APIs
- Shot lookup API usage for metadata
- Overlay/slate essential property updates (burn-in style overlays)
- Queue EXR / Queue MP4 / Queue MOV render queue actions
- Queue EXR (review convention)
- Import exported EXR and import renders
- Pull picture lock using seq timecodes
- Increment shot version by PATCH call
- Delivery prep that copies/renames EXR files to delivery folder convention
### External API endpoints used by DCC tools
- GET /api/ext/projects/{projectCode}/episodes
- GET /api/ext/projects/{projectCode}/shots
- GET /api/ext/shots/lookup
- PATCH /api/ext/shots/{shotId}
### Existing render scripts and pipelines
- AE panel render queue automation (external script)
- Nuke connector script creating write-node outputs and shot scripts:
- [VFXReviewConnector.py](VFXReviewConnector.py)
### ffmpeg scripts
- None found in this repository.
### Proxy generation
- Version.proxyUrl field exists in schema.
- No implemented proxy-generation worker/function found.
### Thumbnail generation
- Upload/assignment flows exist.
- No server-side frame-extract thumbnail generator found.
### Metadata extraction
- EDL / picture-tracker CSV metadata parsing implemented:
- [lib/edl-utils.ts](lib/edl-utils.ts)
- [actions/shots.ts](actions/shots.ts)
### Burn-in generation
- Achieved via DCC overlay/slate layers and essential properties in AE/Nuke tool workflows.
- No ffmpeg burn-in path found.
---
## 5. Review System
### Current review workflow
Core status derivation:
- [lib/shot-status.ts](lib/shot-status.ts)
Server action orchestration:
- [actions/versions.ts](actions/versions.ts)
- [actions/approvals.ts](actions/approvals.ts)
- [actions/tasks.ts](actions/tasks.ts)
- [actions/shots.ts](actions/shots.ts)
- [actions/comments.ts](actions/comments.ts)
### Internal review
- Version upload:
- marks previous versions non-latest
- creates new latest version
- moves Task to INTERNAL_REVIEW
- recalculates Shot.status
### Client review
- Tokenized ReviewSession links
- Optional password gate with signed cookie unlock
- Client comment/annotation/approval endpoints
- Share/unshare semantics via shot and version visibility fields
### Task status flow
Observed statuses:
- TODO
- IN_PROGRESS
- INTERNAL_REVIEW
- CLIENT_REVIEW
- CHANGES
- DONE
### Shot status flow
Derived in priority order:
- REVISIONS (any task CHANGES)
- COMPLETE (shotApprovalStatus CLIENT_APPROVED)
- CLIENT_REVIEW / READY_FOR_CLIENT (internally approved + share flag)
- IN_PROGRESS (task TODO/IN_PROGRESS)
- INTERNAL_REVIEW (tasks exist)
- WAITING (no tasks)
### Approval process
- Version-level approvals create Approval rows and update Version.approvalStatus
- Shot-level client actions supported via client approve endpoint and shot actions
Reference doc in repo:
- [Shot task workflow.md](Shot%20task%20workflow.md)
---
## 6. Authentication
### External tools
- /api/ext/* routes authenticate using API_SECRET_KEY via:
- Authorization: Bearer <key>
- x-api-key header
### Client review links
- tokenized route with ReviewSession lookup
- optional password hash validation and signed unlock cookie
### App users
- NextAuth credentials provider
- bcrypt password hash compare
- JWT session strategy
### Middleware behavior
- Route classes allowed through middleware auth gate:
- /api/ext/
- /api/client/
- /api/display/
- /api/files/
- /api/uploadthing
- Per-route auth still enforced in handlers
Sources:
- [auth.ts](auth.ts)
- [auth.config.ts](auth.config.ts)
- [middleware.ts](middleware.ts)
- [lib/review-auth.ts](lib/review-auth.ts)
---
## 7. Existing Background Jobs
### Cron jobs
- None found.
### Queues
- None found.
### Workers
- None found.
### Polling services
- Display devices poll:
- /api/display/events
- /api/dashboard/stats
### Docker containers
- vfxreview app container
- postgres container
- Source: [docker-compose.yml](docker-compose.yml)
### Scheduled tasks
- Startup migration commands in container entrypoint:
- [entrypoint.sh](entrypoint.sh)
- AE panel Prep Delivery launches background PowerShell copy scripts on workstation (outside web app runtime)
---
## 8. After Effects Integration (Implemented Features)
Repo-level references:
- [VFXReviewConnector.md](VFXReviewConnector.md)
- [EXT_API_REFERENCE.md](EXT_API_REFERENCE.md)
Implemented capabilities observed/documented:
- API calls
- episodes/shots listing and shot lookup
- shot version PATCH
- Authentication
- bearer API token in script
- Comp discovery
- shot code extraction from comp names and dropdown selection
- Render queue integration
- EXR Sequence
- REVIEW_PREVIEW (MP4)
- 4444 Tri (MOV)
- EXR review sequence queue variant
- Output path generation
- shot/version naming conventions + export root paths
- Shot lookup
- uses /api/ext/shots/lookup for metadata and naming decisions
- Upload features
- no direct upload-to-web API flow in panel docs/script (render/output handled in DCC/filesystem)
- Burn-in generation
- overlay and slate essential properties
- Review integration
- preview comp build and version increment sync
---
## 9. Configuration
### Environment variables
Primary reference: [.env.example](.env.example)
Observed vars in code/docs include:
- DATABASE_URL
- NEXTAUTH_SECRET
- NEXTAUTH_URL
- NEXT_PUBLIC_APP_URL
- NEXT_PUBLIC_APP_NAME
- API_SECRET_KEY
- AUTH_SECRET
- STORAGE_PROVIDER
- LOCAL_UPLOAD_DIR
- AWS_ACCESS_KEY_ID
- AWS_SECRET_ACCESS_KEY
- AWS_REGION
- AWS_BUCKET_NAME
- R2_ACCESS_KEY_ID
- R2_SECRET_ACCESS_KEY
- R2_ACCOUNT_ID
- R2_BUCKET_NAME
- R2_PUBLIC_URL
- B2_APPLICATION_KEY_ID
- B2_APPLICATION_KEY
- B2_BUCKET_NAME
- B2_ENDPOINT
- MINIO_ENDPOINT
- MINIO_ACCESS_KEY
- MINIO_SECRET_KEY
- MINIO_BUCKET_NAME
- HETZNER_ENDPOINT
- HETZNER_ACCESS_KEY
- HETZNER_SECRET_KEY
- HETZNER_BUCKET_NAME
- UPLOADTHING_SECRET
- UPLOADTHING_APP_ID
- EMAIL_FROM
- EMAIL_SERVER_HOST
- EMAIL_SERVER_PORT
- EMAIL_SERVER_USER
- EMAIL_SERVER_PASSWORD
- SLACK_DEFAULT_WEBHOOK
### Storage configuration
- Provider abstraction in [lib/storage.ts](lib/storage.ts)
- Hetzner overrides in [actions/settings.ts](actions/settings.ts) and SystemConfig table
### Render configuration
- No server-side renderer configuration module found.
- DCC render templates are documented in [VFXReviewConnector.md](VFXReviewConnector.md).
### ffmpeg configuration
- None found in repository.
### Object storage configuration
- Implemented for AWS S3/R2/B2/MinIO + dedicated Hetzner helper path
---
## 10. Existing Utility Functions (Reusable)
### Metadata extraction
- parseEdlCsv
- parsePictureTrackerCsv
- Source: [lib/edl-utils.ts](lib/edl-utils.ts)
### File scanning
- Local upload tree walker in migration route
- Nuke connector plate/render directory scanners
- Sources:
- [app/api/admin/migration/route.ts](app/api/admin/migration/route.ts)
- [VFXReviewConnector.py](VFXReviewConnector.py)
### EXR sequence detection
- Nuke connector helper sequence detectors and pattern conversion
- Source: [VFXReviewConnector.py](VFXReviewConnector.py)
### MOV generation
- No server-side MOV generation utility found.
- MOV outputs are DCC render queue outputs in external scripts/docs.
### Checksums
- No checksum utility found.
### Frame counting
- durationToFrameCount and frame math helpers
- Source: [lib/frame-utils.ts](lib/frame-utils.ts)
### Timecode extraction/conversion
- frameToTimecode / formatTimecode
- CSV timecode validation/parsing
- Sources:
- [lib/frame-utils.ts](lib/frame-utils.ts)
- [lib/utils.ts](lib/utils.ts)
- [lib/edl-utils.ts](lib/edl-utils.ts)
---
## 11. High-Level Architecture Diagram
```mermaid
flowchart LR
A[AE Panel / Nuke Connector\nExternal DCC tools] -->|API key auth| B[Next.js App Router APIs]
B --> C[Prisma ORM]
C --> D[(PostgreSQL)]
B --> E[Storage Abstraction]
E --> F[(Hetzner Object Storage)]
E --> G[(S3/R2/B2/MinIO)]
B --> H[(Local uploads dir)]
I[Internal reviewers\nNextAuth session] --> B
J[Client reviewers\nToken review sessions] --> B
K[ESP32 display client] -->|x-display-key polling| B
A --> L[Shared filesystem render roots\nEXR/MP4/MOV outputs]
L -->|served/linked via app metadata| B
```
---
## 12. Files Inspected (Primary)
- [prisma/schema.prisma](prisma/schema.prisma)
- [lib/storage.ts](lib/storage.ts)
- [lib/shot-status.ts](lib/shot-status.ts)
- [lib/review-auth.ts](lib/review-auth.ts)
- [lib/edl-utils.ts](lib/edl-utils.ts)
- [lib/frame-utils.ts](lib/frame-utils.ts)
- [lib/utils.ts](lib/utils.ts)
- [auth.ts](auth.ts)
- [auth.config.ts](auth.config.ts)
- [middleware.ts](middleware.ts)
- [next.config.ts](next.config.ts)
- [.env.example](.env.example)
- [docker-compose.yml](docker-compose.yml)
- [Dockerfile](Dockerfile)
- [entrypoint.sh](entrypoint.sh)
- [EXT_API_REFERENCE.md](EXT_API_REFERENCE.md)
- [VFXReviewConnector.md](VFXReviewConnector.md)
- [VFXReviewConnector.py](VFXReviewConnector.py)
- [# VFXReview Connector for Nuke.md](#%20VFXReview%20Connector%20for%20Nuke.md)
- API handlers under [app/api](app/api)
- Server actions under [actions](actions)
---
End of current-state report.
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,273 @@
"use client";
import { useState } from "react";
import Link from "next/link";
import { useQuery, useQueryClient } from "@tanstack/react-query";
import { useSession } from "next-auth/react";
import { RefreshCw, RotateCcw, XCircle, CheckCircle2, Server } from "lucide-react";
import { Button } from "@/components/ui/button";
import { Progress } from "@/components/ui/progress";
import {
Select,
SelectContent,
SelectItem,
SelectTrigger,
SelectValue,
} from "@/components/ui/select";
import { useToast } from "@/components/ui/use-toast";
import { cn } from "@/lib/utils";
import {
EXPORT_STATUS_STYLES,
FAILED_STATUSES,
ACTIVE_STATUSES,
statusLabel,
formatEta,
} from "./status-colors";
interface QueueExport {
id: string;
shotCode: string;
episode: string | null;
projectName: string;
projectCode: string;
versionString: string;
status: string;
statusChangedAt: string;
createdAt: string;
outputDir: string;
job: {
id: string;
attempt: number;
status: string;
progress: number;
currentFrame: number | null;
totalFrames: number | null;
etaSeconds: number | null;
priority: number;
machineName: string | null;
errorMessage: string | null;
} | null;
}
interface Props {
projects: { id: string; name: string; code: string }[];
}
export function PipelineQueueClient({ projects }: Props) {
const { data: session } = useSession();
const { toast } = useToast();
const queryClient = useQueryClient();
const [projectId, setProjectId] = useState<string>("all");
const [statusFilter, setStatusFilter] = useState<string>("all");
const isAdmin = ["ADMIN", "PRODUCER", "SUPERVISOR"].includes(session?.user?.role ?? "");
const { data, isLoading, refetch, isFetching } = useQuery({
queryKey: ["pipeline-queue", projectId, statusFilter],
queryFn: async () => {
const params = new URLSearchParams({ limit: "200" });
if (projectId !== "all") params.set("projectId", projectId);
if (statusFilter !== "all") params.set("status", statusFilter);
const res = await fetch(`/api/render/queue?${params}`);
if (!res.ok) throw new Error("Failed to load queue");
return res.json() as Promise<{ exports: QueueExport[]; pagination: { total: number } }>;
},
refetchInterval: 5000,
});
const exports = data?.exports ?? [];
const counts = {
queued: exports.filter((e) => e.status === "QUEUED").length,
rendering: exports.filter((e) => e.status === "RENDERING").length,
failed: exports.filter((e) => FAILED_STATUSES.includes(e.status)).length,
readyForQc: exports.filter((e) => e.status === "READY_FOR_QC").length,
readyForDelivery: exports.filter((e) => e.status === "READY_FOR_DELIVERY").length,
};
async function action(exportId: string, verb: "retry" | "cancel" | "mark-done") {
const res = await fetch(`/api/render/exports/${exportId}/${verb}`, { method: "POST" });
const body = await res.json().catch(() => ({}));
if (!res.ok) {
toast({ title: `${verb} failed`, description: body.error ?? res.statusText, variant: "destructive" });
} else {
toast({ title: `Export ${verb === "mark-done" ? "marked done" : verb === "retry" ? "requeued" : "cancelled"}` });
}
queryClient.invalidateQueries({ queryKey: ["pipeline-queue"] });
}
const cards: { label: string; value: number; className?: string }[] = [
{ label: "Queued", value: counts.queued },
{ label: "Rendering", value: counts.rendering, className: "text-blue-400" },
{ label: "Failed", value: counts.failed, className: "text-red-400" },
{ label: "Ready for QC", value: counts.readyForQc, className: "text-amber-400" },
{ label: "Ready for Delivery", value: counts.readyForDelivery, className: "text-green-400" },
];
return (
<div className="p-6 space-y-6">
<div className="flex items-center justify-between gap-4 flex-wrap">
<div>
<h1 className="text-2xl font-semibold text-white">Render Queue</h1>
<p className="text-sm text-zinc-400 mt-1">
Exports queued from the AE panel live status, retries and history
</p>
</div>
<div className="flex items-center gap-2">
<Link href="/pipeline/machines">
<Button variant="outline" size="sm">
<Server className="h-4 w-4 mr-2" />
Machines
</Button>
</Link>
<Button variant="outline" size="sm" onClick={() => refetch()} disabled={isFetching}>
<RefreshCw className={cn("h-4 w-4", isFetching && "animate-spin")} />
</Button>
</div>
</div>
{/* Summary cards */}
<div className="grid grid-cols-2 sm:grid-cols-5 gap-3">
{cards.map((c) => (
<div key={c.label} className="rounded-lg border border-zinc-800 bg-zinc-900 px-4 py-3">
<div className={cn("text-2xl font-semibold text-white", c.className)}>{c.value}</div>
<div className="text-xs text-zinc-400 mt-0.5">{c.label}</div>
</div>
))}
</div>
{/* Filters */}
<div className="flex items-center gap-3 flex-wrap">
<Select value={projectId} onValueChange={setProjectId}>
<SelectTrigger className="w-56">
<SelectValue placeholder="All projects" />
</SelectTrigger>
<SelectContent>
<SelectItem value="all">All projects</SelectItem>
{projects.map((p) => (
<SelectItem key={p.id} value={p.id}>
{p.name}
</SelectItem>
))}
</SelectContent>
</Select>
<Select value={statusFilter} onValueChange={setStatusFilter}>
<SelectTrigger className="w-56">
<SelectValue placeholder="All statuses" />
</SelectTrigger>
<SelectContent>
<SelectItem value="all">All statuses</SelectItem>
{Object.keys(EXPORT_STATUS_STYLES).map((s) => (
<SelectItem key={s} value={s}>
{statusLabel(s)}
</SelectItem>
))}
</SelectContent>
</Select>
</div>
{/* Table */}
<div className="rounded-lg border border-zinc-800 overflow-x-auto">
<table className="w-full text-sm">
<thead>
<tr className="border-b border-zinc-800 bg-zinc-900/60 text-left text-xs text-zinc-400">
<th className="px-4 py-2.5 font-medium">Shot</th>
<th className="px-4 py-2.5 font-medium">Version</th>
<th className="px-4 py-2.5 font-medium">Status</th>
<th className="px-4 py-2.5 font-medium w-48">Progress</th>
<th className="px-4 py-2.5 font-medium">Machine</th>
<th className="px-4 py-2.5 font-medium">Attempt</th>
<th className="px-4 py-2.5 font-medium">Queued</th>
<th className="px-4 py-2.5 font-medium text-right">Actions</th>
</tr>
</thead>
<tbody>
{isLoading ? (
<tr>
<td colSpan={8} className="px-4 py-10 text-center text-zinc-500">
Loading
</td>
</tr>
) : exports.length === 0 ? (
<tr>
<td colSpan={8} className="px-4 py-10 text-center text-zinc-500">
No exports yet queue one from the After Effects panel
</td>
</tr>
) : (
exports.map((e) => (
<tr key={e.id} className="border-b border-zinc-800/60 hover:bg-zinc-900/40">
<td className="px-4 py-2.5">
<Link href={`/pipeline/exports/${e.id}`} className="text-white hover:text-amber-400 font-medium">
{e.shotCode}
</Link>
<div className="text-xs text-zinc-500">
{e.projectCode}
{e.episode ? ` · ep ${e.episode}` : ""}
</div>
</td>
<td className="px-4 py-2.5 text-zinc-300">{e.versionString}</td>
<td className="px-4 py-2.5">
<span
className={cn(
"inline-block rounded-full border px-2 py-0.5 text-xs whitespace-nowrap",
EXPORT_STATUS_STYLES[e.status] ?? "bg-zinc-500/15 text-zinc-300 border-zinc-500/30"
)}
title={e.job?.errorMessage ?? undefined}
>
{statusLabel(e.status)}
</span>
</td>
<td className="px-4 py-2.5">
{e.status === "RENDERING" && e.job ? (
<div className="space-y-1">
<Progress value={e.job.progress * 100} className="h-1.5" />
<div className="text-xs text-zinc-500">
{e.job.currentFrame != null && e.job.totalFrames != null
? `frame ${e.job.currentFrame}/${e.job.totalFrames} · `
: ""}
ETA {formatEta(e.job.etaSeconds)}
</div>
</div>
) : (
<span className="text-zinc-600 text-xs"></span>
)}
</td>
<td className="px-4 py-2.5 text-zinc-400">{e.job?.machineName ?? "—"}</td>
<td className="px-4 py-2.5 text-zinc-400">
{e.job ? `${e.job.attempt}` : "—"}
</td>
<td className="px-4 py-2.5 text-zinc-500 text-xs whitespace-nowrap">
{new Date(e.createdAt).toLocaleString()}
</td>
<td className="px-4 py-2.5">
<div className="flex items-center justify-end gap-1">
{FAILED_STATUSES.includes(e.status) && (
<Button variant="ghost" size="icon-sm" title="Retry" onClick={() => action(e.id, "retry")}>
<RotateCcw className="h-4 w-4 text-amber-400" />
</Button>
)}
{ACTIVE_STATUSES.includes(e.status) && (
<Button variant="ghost" size="icon-sm" title="Cancel" onClick={() => action(e.id, "cancel")}>
<XCircle className="h-4 w-4 text-red-400" />
</Button>
)}
{isAdmin && ["QUEUED", "RENDERING"].includes(e.status) && (
<Button
variant="ghost"
size="icon-sm"
title="Mark done manually (rendered outside the pipeline)"
onClick={() => action(e.id, "mark-done")}
>
<CheckCircle2 className="h-4 w-4 text-green-400" />
</Button>
)}
</div>
</td>
</tr>
))
)}
</tbody>
</table>
</div>
</div>
);
}
@@ -0,0 +1,256 @@
"use client";
import Link from "next/link";
import { useQuery, useQueryClient } from "@tanstack/react-query";
import { ArrowLeft, RotateCcw, XCircle } from "lucide-react";
import { Button } from "@/components/ui/button";
import { useToast } from "@/components/ui/use-toast";
import { cn } from "@/lib/utils";
import {
EXPORT_STATUS_STYLES,
FAILED_STATUSES,
ACTIVE_STATUSES,
statusLabel,
} from "../../status-colors";
interface ExportDetail {
export: {
id: string;
status: string;
versionString: string;
statusChangedAt: string;
createdAt: string;
aepPath: string;
compName: string;
outputDir: string;
outputPattern: string;
frameStart: number;
frameEnd: number;
fps: number;
width: number;
height: number;
colorspace: string | null;
exrFileCount: number | null;
exrTotalBytes: number | null;
vfxScope: string | null;
submissionNote: string | null;
submittedByName: string | null;
shot: {
id: string;
shotCode: string;
episode: string | null;
project: { id: string; name: string; code: string };
};
renderJobs: {
id: string;
attempt: number;
status: string;
priority: number;
progress: number;
currentFrame: number | null;
totalFrames: number | null;
errorMessage: string | null;
logTail: string | null;
exitCode: number | null;
claimedAt: string | null;
startedAt: string | null;
finishedAt: string | null;
machine: { id: string; name: string } | null;
}[];
events: {
id: string;
fromStatus: string | null;
toStatus: string;
actorType: string;
actorId: string | null;
note: string | null;
createdAt: string;
}[];
};
}
export function ExportDetailClient({ exportId }: { exportId: string }) {
const { toast } = useToast();
const queryClient = useQueryClient();
const { data, isLoading, error } = useQuery({
queryKey: ["pipeline-export", exportId],
queryFn: async () => {
const res = await fetch(`/api/render/exports/${exportId}`);
if (!res.ok) throw new Error((await res.json().catch(() => ({})))?.error ?? "Failed to load export");
return res.json() as Promise<ExportDetail>;
},
refetchInterval: 5000,
});
async function action(verb: "retry" | "cancel") {
const res = await fetch(`/api/render/exports/${exportId}/${verb}`, { method: "POST" });
const body = await res.json().catch(() => ({}));
if (!res.ok) {
toast({ title: `${verb} failed`, description: body.error ?? res.statusText, variant: "destructive" });
}
queryClient.invalidateQueries({ queryKey: ["pipeline-export", exportId] });
}
if (isLoading) return <div className="p-6 text-zinc-500">Loading</div>;
if (error || !data) {
return (
<div className="p-6 text-red-400">
{(error as Error)?.message ?? "Export not found"}
</div>
);
}
const e = data.export;
const manifestRows: [string, string][] = [
["Project", `${e.shot.project.name} (${e.shot.project.code})`],
["Shot", e.shot.shotCode],
["Comp", e.compName],
["AEP", e.aepPath],
["Output dir", e.outputDir],
["Pattern", e.outputPattern],
["Frames", `${e.frameStart}${e.frameEnd} @ ${e.fps} fps`],
["Resolution", `${e.width}×${e.height}`],
["Colorspace", e.colorspace ?? "—"],
["EXR files", e.exrFileCount != null ? String(e.exrFileCount) : "—"],
["Submitted by", e.submittedByName ?? "—"],
];
return (
<div className="p-6 space-y-6 max-w-5xl">
<div className="flex items-center justify-between gap-4 flex-wrap">
<div className="flex items-center gap-3">
<Link href="/pipeline">
<Button variant="ghost" size="icon-sm">
<ArrowLeft className="h-4 w-4" />
</Button>
</Link>
<div>
<h1 className="text-xl font-semibold text-white">
{e.shot.shotCode} · {e.versionString}
</h1>
<span
className={cn(
"inline-block mt-1 rounded-full border px-2 py-0.5 text-xs",
EXPORT_STATUS_STYLES[e.status] ?? ""
)}
>
{statusLabel(e.status)}
</span>
</div>
</div>
<div className="flex items-center gap-2">
{FAILED_STATUSES.includes(e.status) && (
<Button variant="outline" size="sm" onClick={() => action("retry")}>
<RotateCcw className="h-4 w-4 mr-2 text-amber-400" /> Retry
</Button>
)}
{ACTIVE_STATUSES.includes(e.status) && (
<Button variant="outline" size="sm" onClick={() => action("cancel")}>
<XCircle className="h-4 w-4 mr-2 text-red-400" /> Cancel
</Button>
)}
</div>
</div>
{/* Manifest */}
<div className="rounded-lg border border-zinc-800 bg-zinc-900/40">
<div className="px-4 py-2.5 border-b border-zinc-800 text-sm font-medium text-white">Manifest</div>
<dl className="grid sm:grid-cols-2 gap-x-8 gap-y-2 p-4 text-sm">
{manifestRows.map(([k, v]) => (
<div key={k} className="flex gap-3">
<dt className="w-28 shrink-0 text-zinc-500">{k}</dt>
<dd className="text-zinc-300 break-all">{v}</dd>
</div>
))}
</dl>
</div>
{/* Per-submission slate fields */}
{(e.vfxScope || e.submissionNote) && (
<div className="rounded-lg border border-zinc-800 bg-zinc-900/40">
<div className="px-4 py-2.5 border-b border-zinc-800 text-sm font-medium text-white">
Submission
</div>
<dl className="p-4 space-y-3 text-sm">
{e.vfxScope && (
<div>
<dt className="text-zinc-500 mb-0.5">VFX Scope</dt>
<dd className="text-zinc-300 whitespace-pre-wrap">{e.vfxScope}</dd>
</div>
)}
{e.submissionNote && (
<div>
<dt className="text-zinc-500 mb-0.5">Submission Note</dt>
<dd className="text-zinc-300 whitespace-pre-wrap">{e.submissionNote}</dd>
</div>
)}
</dl>
</div>
)}
{/* Render attempts */}
<div className="rounded-lg border border-zinc-800 bg-zinc-900/40">
<div className="px-4 py-2.5 border-b border-zinc-800 text-sm font-medium text-white">
Render attempts
</div>
<div className="divide-y divide-zinc-800/60">
{e.renderJobs.map((j) => (
<div key={j.id} className="p-4 text-sm">
<div className="flex items-center gap-3 flex-wrap">
<span className="text-white font-medium">Attempt {j.attempt}</span>
<span className="text-xs text-zinc-400">{j.status}</span>
<span className="text-xs text-zinc-500">{j.machine?.name ?? "unclaimed"}</span>
{j.exitCode != null && <span className="text-xs text-zinc-500">exit {j.exitCode}</span>}
{j.startedAt && (
<span className="text-xs text-zinc-500">
started {new Date(j.startedAt).toLocaleString()}
</span>
)}
{j.finishedAt && (
<span className="text-xs text-zinc-500">
finished {new Date(j.finishedAt).toLocaleString()}
</span>
)}
</div>
{j.errorMessage && <div className="mt-1 text-xs text-red-400">{j.errorMessage}</div>}
{j.logTail && (
<details className="mt-2">
<summary className="text-xs text-zinc-500 cursor-pointer hover:text-zinc-300">
Log tail
</summary>
<pre className="mt-2 max-h-64 overflow-auto rounded bg-black/40 p-3 text-xs text-zinc-400 whitespace-pre-wrap">
{j.logTail}
</pre>
</details>
)}
</div>
))}
{e.renderJobs.length === 0 && (
<div className="p-4 text-sm text-zinc-500">No attempts yet</div>
)}
</div>
</div>
{/* Event timeline */}
<div className="rounded-lg border border-zinc-800 bg-zinc-900/40">
<div className="px-4 py-2.5 border-b border-zinc-800 text-sm font-medium text-white">Timeline</div>
<ol className="p-4 space-y-2">
{e.events.map((ev) => (
<li key={ev.id} className="flex items-start gap-3 text-sm">
<span className="text-xs text-zinc-500 w-40 shrink-0 whitespace-nowrap">
{new Date(ev.createdAt).toLocaleString()}
</span>
<span className="text-zinc-300">
{ev.fromStatus ? `${statusLabel(ev.fromStatus)}` : ""}
{statusLabel(ev.toStatus)}
<span className="text-zinc-500"> · {ev.actorType.toLowerCase()}</span>
{ev.note && <span className="block text-xs text-zinc-500">{ev.note}</span>}
</span>
</li>
))}
</ol>
</div>
</div>
);
}
@@ -0,0 +1,17 @@
import { auth } from "@/auth";
import { redirect } from "next/navigation";
import { ExportDetailClient } from "./ExportDetailClient";
export const dynamic = "force-dynamic";
export default async function ExportDetailPage({
params,
}: {
params: Promise<{ exportId: string }>;
}) {
const session = await auth();
if (!session?.user) redirect("/login");
if (session.user.role === "CLIENT") redirect("/dashboard");
const { exportId } = await params;
return <ExportDetailClient exportId={exportId} />;
}
@@ -0,0 +1,206 @@
"use client";
import Link from "next/link";
import { useQuery, useQueryClient } from "@tanstack/react-query";
import { useSession } from "next-auth/react";
import { ArrowLeft, Zap, Power, Cpu, HardDrive, MemoryStick } from "lucide-react";
import { Button } from "@/components/ui/button";
import { Progress } from "@/components/ui/progress";
import { useToast } from "@/components/ui/use-toast";
import { cn } from "@/lib/utils";
interface MachineRow {
id: string;
name: string;
hostname: string;
enabled: boolean;
status: "ONLINE" | "OFFLINE" | "DISABLED";
lastSeenAt: string | null;
workerVersion: string | null;
aeVersion: string | null;
availability: { mode?: string } | null;
renderNowUntil: string | null;
latestHeartbeat: {
createdAt: string;
cpuPercent: number | null;
memPercent: number | null;
diskFreeGb: number | null;
} | null;
currentJob: {
id: string;
exportId: string | null;
shotCode: string | null;
versionString: string | null;
progress: number;
etaSeconds: number | null;
} | null;
}
const STATUS_STYLES: Record<string, string> = {
ONLINE: "bg-green-500/15 text-green-400 border-green-500/30",
OFFLINE: "bg-zinc-500/15 text-zinc-400 border-zinc-500/30",
DISABLED: "bg-red-500/15 text-red-400 border-red-500/30",
};
export function MachinesClient() {
const { data: session } = useSession();
const { toast } = useToast();
const queryClient = useQueryClient();
const isAdmin = ["ADMIN", "PRODUCER", "SUPERVISOR"].includes(session?.user?.role ?? "");
const { data, isLoading } = useQuery({
queryKey: ["pipeline-machines"],
queryFn: async () => {
const res = await fetch("/api/machines");
if (!res.ok) throw new Error("Failed to load machines");
return res.json() as Promise<{ machines: MachineRow[] }>;
},
refetchInterval: 10000,
});
async function patch(machineId: string, body: Record<string, unknown>, okMsg: string) {
const res = await fetch(`/api/machines/${machineId}`, {
method: "PATCH",
headers: { "Content-Type": "application/json" },
body: JSON.stringify(body),
});
const resBody = await res.json().catch(() => ({}));
if (!res.ok) {
toast({ title: "Update failed", description: resBody.error ?? res.statusText, variant: "destructive" });
} else {
toast({ title: okMsg });
}
queryClient.invalidateQueries({ queryKey: ["pipeline-machines"] });
}
const machines = data?.machines ?? [];
return (
<div className="p-6 space-y-6">
<div className="flex items-center gap-3">
<Link href="/pipeline">
<Button variant="ghost" size="icon-sm">
<ArrowLeft className="h-4 w-4" />
</Button>
</Link>
<div>
<h1 className="text-2xl font-semibold text-white">Render Machines</h1>
<p className="text-sm text-zinc-400 mt-1">
Worker status, render windows and the Render Now override
</p>
</div>
</div>
{isLoading ? (
<div className="text-zinc-500">Loading</div>
) : machines.length === 0 ? (
<div className="rounded-lg border border-zinc-800 bg-zinc-900/40 p-10 text-center text-zinc-500">
No machines registered yet install the RenderWorker service on a workstation and it will
appear here after its first registration.
</div>
) : (
<div className="grid gap-4 sm:grid-cols-2 xl:grid-cols-3">
{machines.map((m) => {
const renderNowActive = m.renderNowUntil && new Date(m.renderNowUntil) > new Date();
return (
<div key={m.id} className="rounded-lg border border-zinc-800 bg-zinc-900 p-4 space-y-3">
<div className="flex items-center justify-between">
<div>
<div className="text-white font-medium">{m.name}</div>
<div className="text-xs text-zinc-500">{m.hostname}</div>
</div>
<span
className={cn("rounded-full border px-2 py-0.5 text-xs", STATUS_STYLES[m.status])}
>
{m.status}
</span>
</div>
<div className="text-xs text-zinc-500 space-y-0.5">
<div>Worker {m.workerVersion ?? "?"} · AE {m.aeVersion ?? "?"}</div>
<div>
Availability: {m.availability?.mode ?? "ALWAYS"}
{renderNowActive && (
<span className="text-amber-400">
{" "}· Render Now until {new Date(m.renderNowUntil!).toLocaleTimeString()}
</span>
)}
</div>
{m.lastSeenAt && <div>Last seen {new Date(m.lastSeenAt).toLocaleString()}</div>}
</div>
{m.latestHeartbeat && (
<div className="flex items-center gap-4 text-xs text-zinc-400">
<span className="flex items-center gap-1">
<Cpu className="h-3.5 w-3.5" />
{m.latestHeartbeat.cpuPercent != null ? `${Math.round(m.latestHeartbeat.cpuPercent)}%` : "—"}
</span>
<span className="flex items-center gap-1">
<MemoryStick className="h-3.5 w-3.5" />
{m.latestHeartbeat.memPercent != null ? `${Math.round(m.latestHeartbeat.memPercent)}%` : "—"}
</span>
<span className="flex items-center gap-1">
<HardDrive className="h-3.5 w-3.5" />
{m.latestHeartbeat.diskFreeGb != null ? `${Math.round(m.latestHeartbeat.diskFreeGb)} GB free` : "—"}
</span>
</div>
)}
{m.currentJob ? (
<div className="rounded bg-zinc-800/60 p-2.5 space-y-1.5">
<div className="text-xs text-zinc-300">
Rendering{" "}
{m.currentJob.exportId ? (
<Link
href={`/pipeline/exports/${m.currentJob.exportId}`}
className="text-amber-400 hover:underline"
>
{m.currentJob.shotCode} {m.currentJob.versionString}
</Link>
) : (
"job"
)}
</div>
<Progress value={m.currentJob.progress * 100} className="h-1.5" />
</div>
) : (
<div className="text-xs text-zinc-600">Idle</div>
)}
<div className="flex items-center gap-2 pt-1">
<Button
variant="outline"
size="sm"
disabled={!m.enabled}
onClick={() =>
patch(
m.id,
{ renderNowHours: renderNowActive ? 0 : 4 },
renderNowActive ? "Render Now cleared" : "Render Now active for 4 h"
)
}
>
<Zap className={cn("h-4 w-4 mr-1.5", renderNowActive ? "text-amber-400" : "")} />
{renderNowActive ? "Stop Render Now" : "Render Now (4 h)"}
</Button>
{isAdmin && (
<Button
variant="outline"
size="sm"
onClick={() =>
patch(m.id, { enabled: !m.enabled }, m.enabled ? "Machine disabled" : "Machine enabled")
}
>
<Power className={cn("h-4 w-4 mr-1.5", m.enabled ? "text-red-400" : "text-green-400")} />
{m.enabled ? "Disable" : "Enable"}
</Button>
)}
</div>
</div>
);
})}
</div>
)}
</div>
);
}
@@ -0,0 +1,12 @@
import { auth } from "@/auth";
import { redirect } from "next/navigation";
import { MachinesClient } from "./MachinesClient";
export const dynamic = "force-dynamic";
export default async function MachinesPage() {
const session = await auth();
if (!session?.user) redirect("/login");
if (session.user.role === "CLIENT") redirect("/dashboard");
return <MachinesClient />;
}
+20
View File
@@ -0,0 +1,20 @@
import { auth } from "@/auth";
import { redirect } from "next/navigation";
import { db } from "@/lib/db";
import { PipelineQueueClient } from "./PipelineQueueClient";
export const dynamic = "force-dynamic";
export default async function PipelinePage() {
const session = await auth();
if (!session?.user) redirect("/login");
if (session.user.role === "CLIENT") redirect("/dashboard");
const projects = await db.project.findMany({
where: { status: { in: ["ACTIVE", "ON_HOLD"] } },
select: { id: true, name: true, code: true },
orderBy: { name: "asc" },
});
return <PipelineQueueClient projects={projects} />;
}
+33
View File
@@ -0,0 +1,33 @@
/** Shared status→style maps for the pipeline pages. */
export const EXPORT_STATUS_STYLES: Record<string, string> = {
QUEUED: "bg-zinc-500/15 text-zinc-300 border-zinc-500/30",
RENDERING: "bg-blue-500/15 text-blue-400 border-blue-500/30",
RENDER_FAILED: "bg-red-500/15 text-red-400 border-red-500/30",
VALIDATING: "bg-sky-500/15 text-sky-400 border-sky-500/30",
VALIDATION_FAILED: "bg-red-500/15 text-red-400 border-red-500/30",
GENERATING_PREVIEW: "bg-indigo-500/15 text-indigo-400 border-indigo-500/30",
PREVIEW_FAILED: "bg-red-500/15 text-red-400 border-red-500/30",
READY_FOR_QC: "bg-amber-500/15 text-amber-400 border-amber-500/30",
QC_FAILED: "bg-red-500/15 text-red-400 border-red-500/30",
READY_FOR_DELIVERY: "bg-green-500/15 text-green-400 border-green-500/30",
PACKAGED: "bg-emerald-500/15 text-emerald-400 border-emerald-500/30",
DELIVERED: "bg-emerald-500/15 text-emerald-300 border-emerald-500/30",
SUPERSEDED: "bg-zinc-500/10 text-zinc-500 border-zinc-600/30",
ARCHIVED: "bg-zinc-500/10 text-zinc-500 border-zinc-600/30",
CANCELLED: "bg-zinc-500/10 text-zinc-500 border-zinc-600/30",
};
export const FAILED_STATUSES = ["RENDER_FAILED", "VALIDATION_FAILED", "PREVIEW_FAILED", "QC_FAILED"];
export const ACTIVE_STATUSES = ["QUEUED", "RENDERING", "VALIDATING", "GENERATING_PREVIEW"];
export function statusLabel(status: string): string {
return status.replaceAll("_", " ");
}
export function formatEta(seconds: number | null | undefined): string {
if (seconds == null || seconds <= 0) return "—";
const m = Math.floor(seconds / 60);
const s = Math.round(seconds % 60);
return m > 0 ? `${m}m ${s}s` : `${s}s`;
}
@@ -34,6 +34,7 @@ import {
import { Input } from "@/components/ui/input"; import { Input } from "@/components/ui/input";
import type { ShotWithDetails } from "@/types"; import type { ShotWithDetails } from "@/types";
import { ShotSettingsTab } from "@/components/shots/ShotSettingsTab"; import { ShotSettingsTab } from "@/components/shots/ShotSettingsTab";
import { ShotExportsTab } from "@/components/shots/ShotExportsTab";
import { FootageViewer } from "@/components/shots/FootageViewer"; import { FootageViewer } from "@/components/shots/FootageViewer";
import { HighResUploadDialog } from "@/components/shots/HighResUploadDialog"; import { HighResUploadDialog } from "@/components/shots/HighResUploadDialog";
import { duplicateShot, internallyApproveShot, shareWithClient, unshareFromClient, unapproveShot, updateShotVersion } from "@/actions/shots"; import { duplicateShot, internallyApproveShot, shareWithClient, unshareFromClient, unapproveShot, updateShotVersion } from "@/actions/shots";
@@ -89,7 +90,7 @@ export default function ShotDetailPage() {
const [isDuplicating, setIsDuplicating] = useState(false); const [isDuplicating, setIsDuplicating] = useState(false);
const [isActioning, setIsActioning] = useState(false); const [isActioning, setIsActioning] = useState(false);
const [highResDialogOpen, setHighResDialogOpen] = useState(false); const [highResDialogOpen, setHighResDialogOpen] = useState(false);
const [activeTab, setActiveTab] = useState<"tasks" | "reviews" | "footage" | "settings">("tasks"); const [activeTab, setActiveTab] = useState<"tasks" | "reviews" | "footage" | "exports" | "settings">("tasks");
const [editingVersion, setEditingVersion] = useState(false); const [editingVersion, setEditingVersion] = useState(false);
const [versionInput, setVersionInput] = useState(""); const [versionInput, setVersionInput] = useState("");
const [savingVersion, setSavingVersion] = useState(false); const [savingVersion, setSavingVersion] = useState(false);
@@ -543,6 +544,18 @@ export default function ShotDetailPage() {
<Video className="h-4 w-4" /> <Video className="h-4 w-4" />
Footage Footage
</button> </button>
<button
onClick={() => setActiveTab("exports")}
className={cn(
"flex items-center gap-2 px-4 py-2.5 text-sm font-medium border-b-2 transition-colors -mb-px",
activeTab === "exports"
? "border-amber-500 text-amber-400"
: "border-transparent text-zinc-500 hover:text-zinc-300"
)}
>
<Film className="h-4 w-4" />
Exports
</button>
{canManage && ( {canManage && (
<button <button
onClick={() => setActiveTab("settings")} onClick={() => setActiveTab("settings")}
@@ -658,6 +671,8 @@ export default function ShotDetailPage() {
/> />
)} )}
{activeTab === "exports" && <ShotExportsTab shotId={shot.id} />}
{activeTab === "settings" && canManage && ( {activeTab === "settings" && canManage && (
<ShotSettingsTab shot={shot} artists={artists} onSaved={fetchShot} /> <ShotSettingsTab shot={shot} artists={artists} onSaved={fetchShot} />
)} )}
@@ -0,0 +1,32 @@
import { NextRequest, NextResponse } from "next/server";
import { isExtAuthorized } from "@/lib/render-pipeline/ext-auth";
import { toErrorResponse } from "@/lib/render-pipeline/errors";
import { retryExport } from "@/lib/render-pipeline/exports";
// ── POST /api/ext/exports/{exportId}/retry (E14) ─────────────────────────────
//
// Retry a *_FAILED export from the AE panel: new RenderJob attempt, back to QUEUED.
export async function POST(
req: NextRequest,
{ params }: { params: Promise<{ exportId: string }> }
) {
if (!isExtAuthorized(req)) {
return NextResponse.json({ error: "Unauthorized" }, { status: 401 });
}
const { exportId } = await params;
let note: string | undefined;
try {
const body = await req.json();
if (typeof body?.note === "string") note = body.note;
} catch {
// empty body is fine
}
try {
const result = await retryExport(exportId, { type: "USER", note: note ?? "Retry from AE panel" });
return NextResponse.json(result);
} catch (err) {
const { body, status } = toErrorResponse(err);
return NextResponse.json(body, { status });
}
}
+26
View File
@@ -0,0 +1,26 @@
import { NextRequest, NextResponse } from "next/server";
import { isExtAuthorized } from "@/lib/render-pipeline/ext-auth";
import { toErrorResponse } from "@/lib/render-pipeline/errors";
import { getExportDetail } from "@/lib/render-pipeline/exports";
// ── GET /api/ext/exports/{exportId} (E3) ─────────────────────────────────────
//
// Export detail incl. render attempts and audit events (validations/QC join in
// later phases).
export async function GET(
req: NextRequest,
{ params }: { params: Promise<{ exportId: string }> }
) {
if (!isExtAuthorized(req)) {
return NextResponse.json({ error: "Unauthorized" }, { status: 401 });
}
const { exportId } = await params;
try {
const result = await getExportDetail(exportId);
return NextResponse.json(result);
} catch (err) {
const { body, status } = toErrorResponse(err);
return NextResponse.json(body, { status });
}
}
+28
View File
@@ -0,0 +1,28 @@
import { NextRequest, NextResponse } from "next/server";
import { isExtAuthorized } from "@/lib/render-pipeline/ext-auth";
import { toErrorResponse } from "@/lib/render-pipeline/errors";
import { getLatestExport } from "@/lib/render-pipeline/exports";
// ── GET /api/ext/exports/latest?shotCode=&projectCode= (E2) ──────────────────
//
// Latest export + status for the AE panel status header. Returns
// { "export": null } when the shot has never been exported.
export async function GET(req: NextRequest) {
if (!isExtAuthorized(req)) {
return NextResponse.json({ error: "Unauthorized" }, { status: 401 });
}
const { searchParams } = new URL(req.url);
const shotCode = searchParams.get("shotCode")?.trim();
const projectCode = searchParams.get("projectCode")?.trim();
if (!shotCode) {
return NextResponse.json({ error: "shotCode query param is required" }, { status: 400 });
}
try {
const result = await getLatestExport(shotCode, projectCode);
return NextResponse.json(result);
} catch (err) {
const { body, status } = toErrorResponse(err);
return NextResponse.json(body, { status });
}
}
+47
View File
@@ -0,0 +1,47 @@
import { NextRequest, NextResponse } from "next/server";
import { isExtAuthorized } from "@/lib/render-pipeline/ext-auth";
import { toErrorResponse } from "@/lib/render-pipeline/errors";
import { createExport } from "@/lib/render-pipeline/exports";
// ── POST /api/ext/exports (E1 — Queue Export) ────────────────────────────────
//
// Body: { manifest: RenderManifest, submittedByEmail?, priority?, force? }
// The server decides the new version number, updates Shot.shotVersion/exrOutput,
// supersedes older non-terminal exports, and creates Export(QUEUED) + RenderJob.
export async function POST(req: NextRequest) {
if (!isExtAuthorized(req)) {
return NextResponse.json({ error: "Unauthorized" }, { status: 401 });
}
let body: {
manifest?: unknown;
submittedByEmail?: string;
priority?: number;
force?: boolean;
vfxScope?: string | null;
submissionNote?: string | null;
};
try {
body = await req.json();
} catch {
return NextResponse.json({ error: "Invalid JSON" }, { status: 400 });
}
if (!body.manifest) {
return NextResponse.json({ error: "manifest is required" }, { status: 400 });
}
try {
const result = await createExport({
manifest: body.manifest,
submittedByEmail: body.submittedByEmail ?? null,
priority: typeof body.priority === "number" ? body.priority : undefined,
force: body.force === true,
// undefined inherits the shot's previous export values
vfxScope: body.vfxScope,
submissionNote: body.submissionNote,
});
return NextResponse.json(result, { status: 201 });
} catch (err) {
const { body: errBody, status } = toErrorResponse(err);
return NextResponse.json(errBody, { status });
}
}
@@ -0,0 +1,57 @@
import { NextRequest, NextResponse } from "next/server";
import { randomUUID } from "crypto";
import { isExtAuthorized } from "@/lib/render-pipeline/ext-auth";
import { toErrorResponse, PipelineError } from "@/lib/render-pipeline/errors";
import { db } from "@/lib/db";
import { generateHetznerPresignedUploadUrl, sanitizeFileName } from "@/lib/storage";
// ── POST /api/ext/render/jobs/{jobId}/artifact-presign (E20) ─────────────────
//
// Presigned upload URL for worker artifacts. Kinds map to the existing object
// storage folder layout: preview → videos/, thumbnail → image/,
// metadata/log → renders/.
const KIND_FOLDERS: Record<string, string> = {
preview: "videos",
thumbnail: "image",
metadata: "renders",
log: "renders",
};
export async function POST(
req: NextRequest,
{ params }: { params: Promise<{ jobId: string }> }
) {
if (!isExtAuthorized(req)) {
return NextResponse.json({ error: "Unauthorized" }, { status: 401 });
}
const { jobId } = await params;
let body: { machineId?: string; kind?: string; fileName?: string; contentType?: string };
try {
body = await req.json();
} catch {
return NextResponse.json({ error: "Invalid JSON" }, { status: 400 });
}
try {
if (!body.machineId) throw new PipelineError(400, "machineId is required");
if (!body.fileName) throw new PipelineError(400, "fileName is required");
if (!body.contentType) throw new PipelineError(400, "contentType is required");
const folder = KIND_FOLDERS[body.kind ?? ""];
if (!folder) {
throw new PipelineError(422, `kind must be one of: ${Object.keys(KIND_FOLDERS).join(", ")}`);
}
const job = await db.renderJob.findUnique({ where: { id: jobId }, select: { machineId: true } });
if (!job) throw new PipelineError(404, "Render job not found");
if (job.machineId !== body.machineId) {
throw new PipelineError(409, "Job is not held by this machine (lease reassigned?)");
}
const key = `${folder}/${randomUUID()}-${sanitizeFileName(body.fileName)}`;
const presignedUrl = await generateHetznerPresignedUploadUrl(key, body.contentType);
return NextResponse.json({ presignedUrl, key, url: `/api/files/${key}` });
} catch (err) {
const { body: errBody, status } = toErrorResponse(err);
return NextResponse.json(errBody, { status });
}
}
@@ -0,0 +1,42 @@
import { NextRequest, NextResponse } from "next/server";
import { isExtAuthorized } from "@/lib/render-pipeline/ext-auth";
import { toErrorResponse } from "@/lib/render-pipeline/errors";
import { completeRender } from "@/lib/render-pipeline/jobs";
// ── POST /api/ext/render/jobs/{jobId}/complete-render (E11) ──────────────────
//
// aerender finished with exit 0. Phase 2 interim: Export goes to a provisional
// READY_FOR_QC; Phase 3 inserts VALIDATING between (§17.3).
export async function POST(
req: NextRequest,
{ params }: { params: Promise<{ jobId: string }> }
) {
if (!isExtAuthorized(req)) {
return NextResponse.json({ error: "Unauthorized" }, { status: 401 });
}
const { jobId } = await params;
let body: {
machineId?: string;
renderSeconds?: number;
logFileKey?: string;
logTail?: string;
exrFileCount?: number;
exrTotalBytes?: number;
};
try {
body = await req.json();
} catch {
return NextResponse.json({ error: "Invalid JSON" }, { status: 400 });
}
if (!body.machineId) {
return NextResponse.json({ error: "machineId is required" }, { status: 400 });
}
try {
const result = await completeRender(jobId, body.machineId, body);
return NextResponse.json(result);
} catch (err) {
const { body: errBody, status } = toErrorResponse(err);
return NextResponse.json(errBody, { status });
}
}
@@ -0,0 +1,43 @@
import { NextRequest, NextResponse } from "next/server";
import { isExtAuthorized } from "@/lib/render-pipeline/ext-auth";
import { toErrorResponse } from "@/lib/render-pipeline/errors";
import { reportFail } from "@/lib/render-pipeline/jobs";
// ── POST /api/ext/render/jobs/{jobId}/fail (E10) ─────────────────────────────
//
// Failure report. If retryable and attempts remain the server auto-creates the
// next attempt and returns autoRequeued: true.
export async function POST(
req: NextRequest,
{ params }: { params: Promise<{ jobId: string }> }
) {
if (!isExtAuthorized(req)) {
return NextResponse.json({ error: "Unauthorized" }, { status: 401 });
}
const { jobId } = await params;
let body: {
machineId?: string;
stage?: string;
exitCode?: number;
errorMessage?: string;
logTail?: string;
logFileKey?: string;
retryable?: boolean;
};
try {
body = await req.json();
} catch {
return NextResponse.json({ error: "Invalid JSON" }, { status: 400 });
}
if (!body.machineId) {
return NextResponse.json({ error: "machineId is required" }, { status: 400 });
}
try {
const result = await reportFail(jobId, body.machineId, body);
return NextResponse.json(result);
} catch (err) {
const { body: errBody, status } = toErrorResponse(err);
return NextResponse.json(errBody, { status });
}
}
@@ -0,0 +1,38 @@
import { NextRequest, NextResponse } from "next/server";
import { isExtAuthorized } from "@/lib/render-pipeline/ext-auth";
import { toErrorResponse } from "@/lib/render-pipeline/errors";
import { finalizePreview, type FinalizeInput } from "@/lib/render-pipeline/preview";
// ── POST /api/ext/render/jobs/{jobId}/finalize (E13) ─────────────────────────
//
// Preview artifacts uploaded → register the review MP4 as an internal-only
// Version and move the Export to READY_FOR_QC. Never shares to the client
// portal and never changes task/shot status (§10.0).
export async function POST(
req: NextRequest,
{ params }: { params: Promise<{ jobId: string }> }
) {
if (!isExtAuthorized(req)) {
return NextResponse.json({ error: "Unauthorized" }, { status: 401 });
}
const { jobId } = await params;
let body: FinalizeInput & { machineId?: string };
try {
body = await req.json();
} catch {
return NextResponse.json({ error: "Invalid JSON" }, { status: 400 });
}
if (!body.machineId) {
return NextResponse.json({ error: "machineId is required" }, { status: 400 });
}
try {
const result = await finalizePreview(jobId, body.machineId, body);
return NextResponse.json(result);
} catch (err) {
const { body: errBody, status } = toErrorResponse(err);
return NextResponse.json(errBody, { status });
}
}
@@ -0,0 +1,42 @@
import { NextRequest, NextResponse } from "next/server";
import { isExtAuthorized } from "@/lib/render-pipeline/ext-auth";
import { toErrorResponse } from "@/lib/render-pipeline/errors";
import { reportProgress } from "@/lib/render-pipeline/jobs";
// ── PATCH /api/ext/render/jobs/{jobId}/progress (E9) ─────────────────────────
//
// Progress/ETA report; renews the job lease. Response carries cancelRequested
// so a worker learns about cancellation without any server→worker connection.
export async function PATCH(
req: NextRequest,
{ params }: { params: Promise<{ jobId: string }> }
) {
if (!isExtAuthorized(req)) {
return NextResponse.json({ error: "Unauthorized" }, { status: 401 });
}
const { jobId } = await params;
let body: {
machineId?: string;
progress?: number;
currentFrame?: number;
totalFrames?: number;
etaSeconds?: number;
logTail?: string;
};
try {
body = await req.json();
} catch {
return NextResponse.json({ error: "Invalid JSON" }, { status: 400 });
}
if (!body.machineId) {
return NextResponse.json({ error: "machineId is required" }, { status: 400 });
}
try {
const result = await reportProgress(jobId, body.machineId, body);
return NextResponse.json(result);
} catch (err) {
const { body: errBody, status } = toErrorResponse(err);
return NextResponse.json(errBody, { status });
}
}
+33
View File
@@ -0,0 +1,33 @@
import { NextRequest, NextResponse } from "next/server";
import { isExtAuthorized } from "@/lib/render-pipeline/ext-auth";
import { toErrorResponse } from "@/lib/render-pipeline/errors";
import { claimNextJob } from "@/lib/render-pipeline/jobs";
// ── POST /api/ext/render/jobs/claim (E8) ─────────────────────────────────────
//
// Atomically claim the next queued job (FOR UPDATE SKIP LOCKED). Enforces
// machine availability windows / Render Now / urgent priority server-side
// (§7.10). 204 when nothing is claimable.
export async function POST(req: NextRequest) {
if (!isExtAuthorized(req)) {
return NextResponse.json({ error: "Unauthorized" }, { status: 401 });
}
let body: { machineId?: string; types?: string[] };
try {
body = await req.json();
} catch {
return NextResponse.json({ error: "Invalid JSON" }, { status: 400 });
}
if (!body.machineId) {
return NextResponse.json({ error: "machineId is required" }, { status: 400 });
}
try {
const result = await claimNextJob(body.machineId, body.types);
if (!result) return new NextResponse(null, { status: 204 });
return NextResponse.json(result);
} catch (err) {
const { body: errBody, status } = toErrorResponse(err);
return NextResponse.json(errBody, { status });
}
}
@@ -0,0 +1,33 @@
import { NextRequest, NextResponse } from "next/server";
import { isExtAuthorized } from "@/lib/render-pipeline/ext-auth";
import { toErrorResponse } from "@/lib/render-pipeline/errors";
import { recordHeartbeat } from "@/lib/render-pipeline/machines";
// ── POST /api/ext/workers/{machineId}/heartbeat (E7) ─────────────────────────
//
// Heartbeat + the server→worker command channel: cancellation piggybacks on
// the response (`commands: [{ type: "CANCEL_JOB", jobId }]`) so no server→
// worker connection is ever needed.
export async function POST(
req: NextRequest,
{ params }: { params: Promise<{ machineId: string }> }
) {
if (!isExtAuthorized(req)) {
return NextResponse.json({ error: "Unauthorized" }, { status: 401 });
}
const { machineId } = await params;
let body: { cpuPercent?: number; memPercent?: number; diskFreeGb?: number; currentJobId?: string | null } = {};
try {
body = await req.json();
} catch {
// heartbeat with empty body is fine
}
try {
const result = await recordHeartbeat(machineId, body);
return NextResponse.json(result);
} catch (err) {
const { body: errBody, status } = toErrorResponse(err);
return NextResponse.json(errBody, { status });
}
}
+34
View File
@@ -0,0 +1,34 @@
import { NextRequest, NextResponse } from "next/server";
import { isExtAuthorized } from "@/lib/render-pipeline/ext-auth";
import { toErrorResponse } from "@/lib/render-pipeline/errors";
import { registerMachine } from "@/lib/render-pipeline/machines";
// ── POST /api/ext/workers/register (E6) ──────────────────────────────────────
//
// Idempotent register/upsert on machine name. Returns the server-supplied
// worker config (SystemConfig) so fleet tuning never touches worker installs.
export async function POST(req: NextRequest) {
if (!isExtAuthorized(req)) {
return NextResponse.json({ error: "Unauthorized" }, { status: 401 });
}
let body: {
name?: string;
hostname?: string;
workerVersion?: string;
aeVersion?: string;
capabilities?: unknown;
};
try {
body = await req.json();
} catch {
return NextResponse.json({ error: "Invalid JSON" }, { status: 400 });
}
try {
const result = await registerMachine(body);
return NextResponse.json(result);
} catch (err) {
const { body: errBody, status } = toErrorResponse(err);
return NextResponse.json(errBody, { status });
}
}
+41
View File
@@ -0,0 +1,41 @@
import { NextRequest, NextResponse } from "next/server";
import { toErrorResponse, PipelineError } from "@/lib/render-pipeline/errors";
import { requirePipelineUser, requirePipelineAdmin } from "@/lib/render-pipeline/session-auth";
import { updateMachine, type MachineAvailability } from "@/lib/render-pipeline/machines";
// ── PATCH /api/machines/{machineId} ──────────────────────────────────────────
//
// Body (all optional):
// enabled: boolean — admin kill-switch (admin/producer/supervisor)
// availability: {...}|null — §7.10 schedule config (admin/producer/supervisor)
// renderNowHours: number|0 — "Render Now" override (any studio user);
// 0 or null clears it
export async function PATCH(
req: NextRequest,
{ params }: { params: Promise<{ machineId: string }> }
) {
try {
const { machineId } = await params;
let body: { enabled?: boolean; availability?: MachineAvailability | null; renderNowHours?: number | null };
try {
body = await req.json();
} catch {
throw new PipelineError(400, "Invalid JSON");
}
// Render Now is available to every studio user (§7.10); enable/disable and
// availability windows are admin-level controls.
if (body.enabled !== undefined || body.availability !== undefined) {
await requirePipelineAdmin();
} else {
await requirePipelineUser();
}
const result = await updateMachine(machineId, body);
return NextResponse.json(result);
} catch (err) {
const { body, status } = toErrorResponse(err);
return NextResponse.json(body, { status });
}
}
+20
View File
@@ -0,0 +1,20 @@
import { NextResponse } from "next/server";
import { toErrorResponse } from "@/lib/render-pipeline/errors";
import { requirePipelineUser } from "@/lib/render-pipeline/session-auth";
import { listMachines } from "@/lib/render-pipeline/machines";
// ── GET /api/machines (E21) ──────────────────────────────────────────────────
//
// Machine list for the Machine Monitoring page: derived ONLINE/OFFLINE status,
// latest heartbeat stats, current job.
export async function GET() {
try {
await requirePipelineUser();
const machines = await listMachines();
return NextResponse.json({ machines });
} catch (err) {
const { body, status } = toErrorResponse(err);
return NextResponse.json(body, { status });
}
}
@@ -0,0 +1,21 @@
import { NextRequest, NextResponse } from "next/server";
import { toErrorResponse } from "@/lib/render-pipeline/errors";
import { requirePipelineUser } from "@/lib/render-pipeline/session-auth";
import { cancelExport } from "@/lib/render-pipeline/exports";
// ── POST /api/render/exports/{exportId}/cancel (E15 / T18) ───────────────────
export async function POST(
_req: NextRequest,
{ params }: { params: Promise<{ exportId: string }> }
) {
try {
const user = await requirePipelineUser();
const { exportId } = await params;
const result = await cancelExport(exportId, { type: "USER", id: user.id, note: "Cancelled from web UI" });
return NextResponse.json(result);
} catch (err) {
const { body, status } = toErrorResponse(err);
return NextResponse.json(body, { status });
}
}
@@ -0,0 +1,31 @@
import { NextRequest, NextResponse } from "next/server";
import { toErrorResponse } from "@/lib/render-pipeline/errors";
import { requirePipelineAdmin } from "@/lib/render-pipeline/session-auth";
import { markExportDoneManually } from "@/lib/render-pipeline/exports";
// ── POST /api/render/exports/{exportId}/mark-done ────────────────────────────
//
// Phase 1 stopgap (§17.1): lets the studio keep using the old manual render
// path while the queue is validated. Remove once workers render for real.
export async function POST(
req: NextRequest,
{ params }: { params: Promise<{ exportId: string }> }
) {
try {
const user = await requirePipelineAdmin();
const { exportId } = await params;
let note: string | undefined;
try {
const body = await req.json();
if (typeof body?.note === "string") note = body.note;
} catch {
// empty body is fine
}
const result = await markExportDoneManually(exportId, user.id, note);
return NextResponse.json(result);
} catch (err) {
const { body, status } = toErrorResponse(err);
return NextResponse.json(body, { status });
}
}
@@ -0,0 +1,21 @@
import { NextRequest, NextResponse } from "next/server";
import { toErrorResponse } from "@/lib/render-pipeline/errors";
import { requirePipelineUser } from "@/lib/render-pipeline/session-auth";
import { retryExport } from "@/lib/render-pipeline/exports";
// ── POST /api/render/exports/{exportId}/retry (E15) ──────────────────────────
export async function POST(
_req: NextRequest,
{ params }: { params: Promise<{ exportId: string }> }
) {
try {
const user = await requirePipelineUser();
const { exportId } = await params;
const result = await retryExport(exportId, { type: "USER", id: user.id, note: "Retry from web UI" });
return NextResponse.json(result);
} catch (err) {
const { body, status } = toErrorResponse(err);
return NextResponse.json(body, { status });
}
}
@@ -0,0 +1,23 @@
import { NextRequest, NextResponse } from "next/server";
import { toErrorResponse } from "@/lib/render-pipeline/errors";
import { requirePipelineUser } from "@/lib/render-pipeline/session-auth";
import { getExportDetail } from "@/lib/render-pipeline/exports";
// ── GET /api/render/exports/{exportId} ───────────────────────────────────────
//
// Export detail for the web Export page (internal mirror of E3).
export async function GET(
_req: NextRequest,
{ params }: { params: Promise<{ exportId: string }> }
) {
try {
await requirePipelineUser();
const { exportId } = await params;
const result = await getExportDetail(exportId);
return NextResponse.json(result);
} catch (err) {
const { body, status } = toErrorResponse(err);
return NextResponse.json(body, { status });
}
}
@@ -0,0 +1,24 @@
import { NextRequest, NextResponse } from "next/server";
import { toErrorResponse, PipelineError } from "@/lib/render-pipeline/errors";
import { requirePipelineUser } from "@/lib/render-pipeline/session-auth";
import { getJobDetail } from "@/lib/render-pipeline/jobs";
import { cancelExport } from "@/lib/render-pipeline/exports";
// ── POST /api/render/jobs/{jobId}/cancel (E15 / T18) ─────────────────────────
export async function POST(
_req: NextRequest,
{ params }: { params: Promise<{ jobId: string }> }
) {
try {
const user = await requirePipelineUser();
const { jobId } = await params;
const job = await getJobDetail(jobId);
if (!job.exportId) throw new PipelineError(422, "Job has no export to cancel");
const result = await cancelExport(job.exportId, { type: "USER", id: user.id, note: "Cancelled from web UI" });
return NextResponse.json(result);
} catch (err) {
const { body, status } = toErrorResponse(err);
return NextResponse.json(body, { status });
}
}
@@ -0,0 +1,26 @@
import { NextRequest, NextResponse } from "next/server";
import { toErrorResponse, PipelineError } from "@/lib/render-pipeline/errors";
import { requirePipelineUser } from "@/lib/render-pipeline/session-auth";
import { getJobDetail } from "@/lib/render-pipeline/jobs";
import { retryExport } from "@/lib/render-pipeline/exports";
// ── POST /api/render/jobs/{jobId}/retry (E15) ────────────────────────────────
//
// Job-level retry resolves to its Export (each retry is a fresh attempt row).
export async function POST(
_req: NextRequest,
{ params }: { params: Promise<{ jobId: string }> }
) {
try {
const user = await requirePipelineUser();
const { jobId } = await params;
const job = await getJobDetail(jobId);
if (!job.exportId) throw new PipelineError(422, "Job has no export to retry");
const result = await retryExport(job.exportId, { type: "USER", id: user.id, note: "Retry from web UI" });
return NextResponse.json(result);
} catch (err) {
const { body, status } = toErrorResponse(err);
return NextResponse.json(body, { status });
}
}
+34
View File
@@ -0,0 +1,34 @@
import { NextRequest, NextResponse } from "next/server";
import { ExportStatus } from "@prisma/client";
import { toErrorResponse } from "@/lib/render-pipeline/errors";
import { requirePipelineUser } from "@/lib/render-pipeline/session-auth";
import { listExports } from "@/lib/render-pipeline/exports";
// ── GET /api/render/queue (E21) ──────────────────────────────────────────────
//
// Exports list for the Render Queue page. Optional filters: projectId,
// episode, status; standard pagination. Defaults to all statuses so the queue
// page can show history too — the UI filters live states client-side.
export async function GET(req: NextRequest) {
try {
await requirePipelineUser();
const { searchParams } = new URL(req.url);
const status = searchParams.get("status") ?? undefined;
const result = await listExports({
projectId: searchParams.get("projectId") ?? undefined,
episode: searchParams.get("episode") ?? undefined,
shotId: searchParams.get("shotId") ?? undefined,
status:
status && (Object.values(ExportStatus) as string[]).includes(status)
? (status as ExportStatus)
: undefined,
page: Number(searchParams.get("page") ?? "1") || 1,
limit: Number(searchParams.get("limit") ?? "50") || 50,
});
return NextResponse.json(result);
} catch (err) {
const { body, status } = toErrorResponse(err);
return NextResponse.json(body, { status });
}
}
+2
View File
@@ -21,6 +21,7 @@ import {
CloudUpload, CloudUpload,
Clapperboard, Clapperboard,
LayoutGrid, LayoutGrid,
Server,
} from 'lucide-react'; } from 'lucide-react';
import { useState } from 'react'; import { useState } from 'react';
import { useSession } from 'next-auth/react'; import { useSession } from 'next-auth/react';
@@ -32,6 +33,7 @@ const navItems = [
{ href: '/shot-status', label: 'Shot Status', icon: BarChart2, hideForClient: true }, { href: '/shot-status', label: 'Shot Status', icon: BarChart2, hideForClient: true },
{ href: '/playlist', label: 'Playlist', icon: ListVideo, hideForClient: true }, { href: '/playlist', label: 'Playlist', icon: ListVideo, hideForClient: true },
{ href: '/tasks', label: 'My Tasks', icon: ListTodo, hideForClient: true }, { href: '/tasks', label: 'My Tasks', icon: ListTodo, hideForClient: true },
{ href: '/pipeline', label: 'Pipeline', icon: Server, hideForClient: true },
{ href: '/shoot-log', label: 'Shot Log', icon: Clapperboard, supervisorOnly: true }, { href: '/shoot-log', label: 'Shot Log', icon: Clapperboard, supervisorOnly: true },
{ href: '/storyboard', label: 'Storyboard', icon: LayoutGrid, supervisorOnly: true }, { href: '/storyboard', label: 'Storyboard', icon: LayoutGrid, supervisorOnly: true },
{ href: '/schedule', label: 'Schedule', icon: CalendarRange, adminOnly: true }, { href: '/schedule', label: 'Schedule', icon: CalendarRange, adminOnly: true },
+85
View File
@@ -0,0 +1,85 @@
"use client";
import Link from "next/link";
import { useQuery } from "@tanstack/react-query";
import { Layers, ExternalLink } from "lucide-react";
import { cn } from "@/lib/utils";
import {
EXPORT_STATUS_STYLES,
statusLabel,
} from "@/app/(dashboard)/pipeline/status-colors";
interface ExportRow {
id: string;
versionString: string;
status: string;
statusChangedAt: string;
createdAt: string;
outputDir: string;
job: {
attempt: number;
progress: number;
machineName: string | null;
errorMessage: string | null;
} | null;
}
/**
* "Exports" tab on the shot detail page (RenderPipeline2 §13) the render
* pipeline history for this shot, linking into the Export detail page.
*/
export function ShotExportsTab({ shotId }: { shotId: string }) {
const { data, isLoading } = useQuery({
queryKey: ["shot-exports", shotId],
queryFn: async () => {
const res = await fetch(`/api/render/queue?shotId=${shotId}&limit=100`);
if (!res.ok) throw new Error("Failed to load exports");
return res.json() as Promise<{ exports: ExportRow[] }>;
},
refetchInterval: 10000,
});
const exports = data?.exports ?? [];
if (isLoading) {
return <div className="py-10 text-center text-sm text-muted-foreground">Loading exports</div>;
}
if (exports.length === 0) {
return (
<div className="flex flex-col items-center justify-center py-16 gap-2 text-muted-foreground">
<Layers className="h-8 w-8 opacity-30" />
<p className="text-sm">No pipeline exports yet queue one from the After Effects panel.</p>
</div>
);
}
return (
<div className="space-y-2">
{exports.map((e) => (
<Link
key={e.id}
href={`/pipeline/exports/${e.id}`}
className="flex items-center gap-4 rounded-lg border border-border bg-card p-4 hover:border-zinc-600 transition-colors"
>
<span className="font-mono text-sm text-white w-14">{e.versionString}</span>
<span
className={cn(
"rounded-full border px-2 py-0.5 text-xs whitespace-nowrap",
EXPORT_STATUS_STYLES[e.status] ?? "bg-zinc-500/15 text-zinc-300 border-zinc-500/30"
)}
>
{statusLabel(e.status)}
</span>
<span className="flex-1 min-w-0 truncate text-xs text-muted-foreground">
{e.job?.machineName ? `${e.job.machineName} · attempt ${e.job.attempt} · ` : ""}
{e.outputDir}
</span>
<span className="text-xs text-muted-foreground whitespace-nowrap">
{new Date(e.createdAt).toLocaleDateString()}
</span>
<ExternalLink className="h-3.5 w-3.5 text-muted-foreground shrink-0" />
</Link>
))}
</div>
);
}
+18
View File
@@ -0,0 +1,18 @@
services:
postgres:
image: postgres:15
container_name: vfxreview-dev-db
environment:
POSTGRES_DB: feedback
POSTGRES_USER: postgres
POSTGRES_PASSWORD: postgres
ports:
- "5433:5432"
volumes:
- postgres-data:/var/lib/postgresql/data
volumes:
postgres-data:
+12
View File
@@ -0,0 +1,12 @@
/**
* Next.js server startup hook starts the render-pipeline reaper
* (RenderPipeline2 §7.7, studio decision 18.1-Q8). The reaper requeues or
* fails jobs whose lease expired, marks silent machines OFFLINE, and prunes
* old heartbeats. This is the only background job the web app runs.
*/
export async function register() {
if (process.env.NEXT_RUNTIME === "nodejs") {
const { startReaper } = await import("@/lib/render-pipeline/reaper");
startReaper();
}
}
+83
View File
@@ -0,0 +1,83 @@
import { db } from "@/lib/db";
/**
* Pipeline-wide settings live in the existing SystemConfig key/value table
* (RenderPipeline2 §14.1) so the fleet can be tuned without redeploys.
*/
export const PIPELINE_CONFIG_DEFAULTS = {
"render.pollSeconds": 10,
"render.heartbeatSeconds": 30,
"render.leaseSeconds": 300,
"render.maxAttempts": 3,
"render.stallTimeoutSeconds": 600,
"render.urgentPriorityThreshold": 20,
"render.outputRoot": "//SAN/renders",
"validation.sampleEvery": 25,
"preview.maxWidth": 1920,
// Preview stage (§9): headless AE rebuild of the render with slate/burn-ins
"preview.enabled": "true",
"preview.templateAep": "X:/shared_projects_2026/UNGO_VFX/production/working_files/UNG_VFXOVERLAY_SLATE_TEMPLATE.aep",
"preview.templateComp": "UNG_EXPORT_TEMPLATE",
"preview.overlayComp": "UNG_VFX_OVERLAY",
"preview.lutComp": "_SHOW LUT",
"preview.movTemplate": "4444 Tri",
"preview.mp4Template": "REVIEW_PREVIEW",
// Essential Property names on the NETFLIX_SLATE layer for the per-submission
// fields. If a name is wrong the build logs the template's actual property
// names as a warning, so it is a config fix rather than a code change.
"preview.slateScopeProp": "Scope",
"preview.slateSubmissionProp": "Notes",
} as const;
export type PipelineConfigKey = keyof typeof PIPELINE_CONFIG_DEFAULTS;
export async function getConfigValue(key: PipelineConfigKey): Promise<string> {
const row = await db.systemConfig.findUnique({ where: { key } });
return row?.value ?? String(PIPELINE_CONFIG_DEFAULTS[key]);
}
export async function getConfigNumber(key: PipelineConfigKey): Promise<number> {
const raw = await getConfigValue(key);
const n = Number(raw);
return Number.isFinite(n) ? n : Number(PIPELINE_CONFIG_DEFAULTS[key]);
}
export async function getConfigBoolean(key: PipelineConfigKey): Promise<boolean> {
const raw = (await getConfigValue(key)).trim().toLowerCase();
return raw === "true" || raw === "1" || raw === "yes";
}
/** Preview-stage settings handed to the worker inside the PREVIEW_ONLY manifest. */
export async function getPreviewConfig() {
const [
enabled, templateAep, templateComp, overlayComp, lutComp, movTemplate, mp4Template,
slateScopeProp, slateSubmissionProp,
] = await Promise.all([
getConfigBoolean("preview.enabled"),
getConfigValue("preview.templateAep"),
getConfigValue("preview.templateComp"),
getConfigValue("preview.overlayComp"),
getConfigValue("preview.lutComp"),
getConfigValue("preview.movTemplate"),
getConfigValue("preview.mp4Template"),
getConfigValue("preview.slateScopeProp"),
getConfigValue("preview.slateSubmissionProp"),
]);
return {
enabled, templateAep, templateComp, overlayComp, lutComp, movTemplate, mp4Template,
slateScopeProp, slateSubmissionProp,
};
}
/** Worker-facing config bundle returned by E6 registration. */
export async function getWorkerConfig() {
const [pollSeconds, heartbeatSeconds, leaseSeconds, maxAttempts, stallTimeoutSeconds] =
await Promise.all([
getConfigNumber("render.pollSeconds"),
getConfigNumber("render.heartbeatSeconds"),
getConfigNumber("render.leaseSeconds"),
getConfigNumber("render.maxAttempts"),
getConfigNumber("render.stallTimeoutSeconds"),
]);
return { pollSeconds, heartbeatSeconds, leaseSeconds, maxAttempts, stallTimeoutSeconds };
}
+26
View File
@@ -0,0 +1,26 @@
/**
* Typed service-layer error carrying an HTTP status so both auth fronts
* (ext API-key routes and internal session routes) map errors identically.
*/
export class PipelineError extends Error {
status: number;
details?: unknown;
constructor(status: number, message: string, details?: unknown) {
super(message);
this.name = "PipelineError";
this.status = status;
this.details = details;
}
}
export function toErrorResponse(err: unknown): { body: { error: string; details?: unknown }; status: number } {
if (err instanceof PipelineError) {
return {
body: { error: err.message, ...(err.details !== undefined ? { details: err.details } : {}) },
status: err.status,
};
}
console.error("[render-pipeline]", err);
return { body: { error: "Internal server error" }, status: 500 };
}
+504
View File
@@ -0,0 +1,504 @@
import { ExportStatus, Prisma } from "@prisma/client";
import { db } from "@/lib/db";
import { PipelineError } from "./errors";
import { renderManifestSchema, type RenderManifest } from "./manifest";
import { applyTransition, type TransitionActor } from "./transitions";
import { getConfigNumber, getConfigValue } from "./config";
/** Statuses in which a shot has a live export in flight (§6.4 E1 409 rule). */
const ACTIVE_STATUSES: ExportStatus[] = [
"QUEUED",
"RENDERING",
"VALIDATING",
"GENERATING_PREVIEW",
];
/** Non-terminal statuses auto-superseded when a newer Export is queued (T1/T17). */
const SUPERSEDABLE_STATUSES: ExportStatus[] = [
"QUEUED",
"RENDER_FAILED",
"VALIDATION_FAILED",
"PREVIEW_FAILED",
"READY_FOR_QC",
"QC_FAILED",
"READY_FOR_DELIVERY",
];
const OPEN_JOB_STATUSES = ["QUEUED", "CLAIMED", "RUNNING"] as const;
export function parseVersionString(v: string | null | undefined): number {
const m = /^v(\d+)$/i.exec(v?.trim() ?? "");
return m ? parseInt(m[1], 10) : 0;
}
export function formatVersionString(n: number): string {
return `v${String(n).padStart(3, "0")}`;
}
/** `UNG_..._cmp_TT_v003` + 4 → `UNG_..._cmp_TT_v004`; falls back to house convention. */
function nextExrOutputBase(current: string | null, shotCode: string, versionString: string): string {
if (current && /_v\d+$/i.test(current)) return current.replace(/_v\d+$/i, `_${versionString}`);
if (current) return `${current}_${versionString}`;
return renderOutputBase(shotCode, versionString);
}
/**
* Rendered EXR naming for the pipeline matches the panel's
* "Queue EXR (Review)" convention (`{shotCode}_cmp_TT_{version}`), which is
* shot-code based and deliberately excludes the source clip name that
* `Shot.exrOutput` carries from the EDL import.
*/
function renderOutputBase(shotCode: string, versionString: string): string {
return `${shotCode}_cmp_TT_${versionString}`;
}
export interface CreateExportInput {
manifest: unknown;
submittedByEmail?: string | null;
priority?: number;
force?: boolean;
/**
* Per-submission slate fields. `undefined` inherits the previous export's
* value for this shot; `null` or "" explicitly clears it.
*/
vfxScope?: string | null;
submissionNote?: string | null;
}
/**
* E1 Queue Export (§6.4). Transactionally: decides the new version number
* (server-side, race-free), updates Shot.shotVersion/exrOutput exactly as the
* panel's legacy PATCH did, supersedes older non-terminal exports, creates
* Export(QUEUED) + RenderJob attempt 1 with the manifest snapshot.
*/
export async function createExport(input: CreateExportInput) {
const parsed = renderManifestSchema.safeParse(input.manifest);
if (!parsed.success) {
throw new PipelineError(422, "Manifest validation error", parsed.error.issues);
}
const manifest = parsed.data;
const shot = await db.shot.findUnique({
where: { id: manifest.shotId },
include: {
project: { select: { id: true, code: true, showId: true } },
tasks: { where: { type: "COMP" }, orderBy: { sortOrder: "asc" }, take: 1, select: { id: true } },
},
});
if (!shot) throw new PipelineError(404, `Shot not found: ${manifest.shotId}`);
// Cross-checks (§6.2): manifest vs DB truth
if (shot.shotCode !== manifest.shotCode) {
throw new PipelineError(422, `Manifest shotCode "${manifest.shotCode}" does not match shot ${shot.id} ("${shot.shotCode}")`);
}
if (shot.project.code !== manifest.projectCode) {
throw new PipelineError(422, `Manifest projectCode "${manifest.projectCode}" does not match shot's project ("${shot.project.code}")`);
}
// A 0/0 shot range means "never imported", not a real single-frame shot at 0
const shotRangeKnown =
shot.frameStart != null && shot.frameEnd != null && !(shot.frameStart === 0 && shot.frameEnd === 0);
if (
!input.force &&
shotRangeKnown &&
(shot.frameStart !== manifest.frameStart || shot.frameEnd !== manifest.frameEnd)
) {
throw new PipelineError(422, `Manifest frame range ${manifest.frameStart}-${manifest.frameEnd} does not match shot record ${shot.frameStart}-${shot.frameEnd} (pass force to override)`);
}
const activeExport = await db.export.findFirst({
where: { shotId: shot.id, status: { in: ACTIVE_STATUSES } },
select: { id: true, status: true, versionString: true },
});
if (activeExport && !input.force) {
throw new PipelineError(
409,
`An active export (${activeExport.versionString}, ${activeExport.status}) already exists for this shot — pass force to supersede it`,
{ activeExportId: activeExport.id }
);
}
const submittedBy = input.submittedByEmail
? await db.user.findUnique({ where: { email: input.submittedByEmail }, select: { id: true, name: true } })
: null;
const [maxAttempts, outputRoot] = await Promise.all([
getConfigNumber("render.maxAttempts"),
getConfigValue("render.outputRoot"),
]);
// Retry loop: @@unique([shotId, versionNumber]) guards the version-increment
// race between two concurrent E1 calls; loser recomputes and retries.
for (let tryNo = 0; tryNo < 3; tryNo++) {
try {
return await db.$transaction(async (tx) => {
const latest = await tx.export.aggregate({
where: { shotId: shot.id },
_max: { versionNumber: true },
});
const freshShot = await tx.shot.findUniqueOrThrow({
where: { id: shot.id },
select: { shotVersion: true, exrOutput: true },
});
// Slate fields carry forward from the shot's previous submission
// unless the caller supplies new ones.
const previous = await tx.export.findFirst({
where: { shotId: shot.id },
orderBy: { versionNumber: "desc" },
select: { vfxScope: true, submissionNote: true },
});
const vfxScope = input.vfxScope !== undefined ? input.vfxScope : (previous?.vfxScope ?? null);
const submissionNote =
input.submissionNote !== undefined ? input.submissionNote : (previous?.submissionNote ?? null);
const versionNumber =
Math.max(latest._max.versionNumber ?? 0, parseVersionString(freshShot.shotVersion)) + 1;
const versionString = formatVersionString(versionNumber);
const exrBase = nextExrOutputBase(freshShot.exrOutput, shot.shotCode, versionString);
const outputDir =
manifest.outputDir === "auto"
? [outputRoot, shot.project.code, ...(shot.episode ? [shot.episode] : []), shot.shotCode, versionString].join("/")
: manifest.outputDir;
const outputPattern =
manifest.outputPattern === "auto"
? `${renderOutputBase(shot.shotCode, versionString)}.[#####].exr`
: manifest.outputPattern;
const resolvedManifest: RenderManifest = { ...manifest, outputDir, outputPattern };
// Supersede older non-terminal exports (T1/T17); cancel their open jobs.
const toSupersede = await tx.export.findMany({
where: {
shotId: shot.id,
status: { in: input.force ? [...SUPERSEDABLE_STATUSES, ...ACTIVE_STATUSES] : SUPERSEDABLE_STATUSES },
},
select: { id: true, status: true },
});
const created = await tx.export.create({
data: {
shotId: shot.id,
projectId: shot.project.id,
taskId: shot.tasks[0]?.id ?? null,
versionNumber,
versionString,
status: "QUEUED",
aepPath: manifest.aepPath,
compName: manifest.compName,
rendererType: manifest.rendererType,
outputDir,
outputPattern,
frameStart: manifest.frameStart,
frameEnd: manifest.frameEnd,
fps: manifest.fps,
width: manifest.width,
height: manifest.height,
colorspace: manifest.expected?.colorspace ?? null,
vfxScope,
submissionNote,
submittedById: submittedBy?.id ?? null,
submittedByName: submittedBy?.name ?? input.submittedByEmail ?? null,
},
});
for (const old of toSupersede) {
await tx.renderJob.updateMany({
where: { exportId: old.id, status: { in: [...OPEN_JOB_STATUSES] } },
data: { status: "CANCELLED", finishedAt: new Date(), errorMessage: `Superseded by export ${created.id} (${versionString})` },
});
await applyTransition(
tx,
old,
"SUPERSEDED",
{ type: "SYSTEM", note: `Superseded by ${versionString}` },
{ supersededById: created.id }
);
}
const renderJob = await tx.renderJob.create({
data: {
type: "AE_RENDER",
exportId: created.id,
attempt: 1,
maxAttempts,
priority: input.priority ?? 50,
manifest: resolvedManifest as unknown as Prisma.InputJsonValue,
},
});
await tx.exportEvent.create({
data: {
exportId: created.id,
fromStatus: null,
toStatus: "QUEUED",
actorType: "USER",
actorId: submittedBy?.id ?? null,
note: `Queue Export ${versionString}${input.force ? " (force)" : ""}`,
},
});
// Mirror the legacy panel PATCH: Shot stays the canonical "current version"
const updatedShot = await tx.shot.update({
where: { id: shot.id },
data: { shotVersion: versionString, exrOutput: exrBase },
select: { id: true, shotVersion: true, exrOutput: true },
});
return {
export: {
id: created.id,
shotCode: shot.shotCode,
versionNumber,
versionString,
status: created.status,
outputDir,
outputPattern,
vfxScope,
submissionNote,
},
renderJob: { id: renderJob.id, attempt: renderJob.attempt, priority: renderJob.priority },
shot: updatedShot,
superseded: toSupersede.map((s) => s.id),
};
});
} catch (err) {
if (err instanceof Prisma.PrismaClientKnownRequestError && err.code === "P2002" && tryNo < 2) {
continue; // concurrent E1 won the version — recompute
}
throw err;
}
}
throw new PipelineError(409, "Could not allocate a version number after retries");
}
function serializeExport<T extends { exrTotalBytes: bigint | null }>(e: T) {
return { ...e, exrTotalBytes: e.exrTotalBytes == null ? null : Number(e.exrTotalBytes) };
}
/** E2 — latest export + status for the AE panel header (§6.4). */
export async function getLatestExport(shotCode: string, projectCode?: string | null) {
const shot = await db.shot.findFirst({
where: { shotCode, ...(projectCode ? { project: { code: projectCode } } : {}) },
select: { id: true },
});
if (!shot) throw new PipelineError(404, "Shot not found");
const exp = await db.export.findFirst({
where: { shotId: shot.id },
orderBy: { versionNumber: "desc" },
include: {
renderJobs: {
orderBy: { attempt: "desc" },
take: 1,
include: { machine: { select: { name: true } } },
},
},
});
if (!exp) return { export: null };
const job = exp.renderJobs[0] ?? null;
return {
export: {
id: exp.id,
versionNumber: exp.versionNumber,
versionString: exp.versionString,
status: exp.status,
statusChangedAt: exp.statusChangedAt,
outputDir: exp.outputDir,
outputPattern: exp.outputPattern,
// Panel prefills its slate fields from these
vfxScope: exp.vfxScope,
submissionNote: exp.submissionNote,
renderJob: job
? {
id: job.id,
attempt: job.attempt,
status: job.status,
progress: job.progress,
currentFrame: job.currentFrame,
totalFrames: job.totalFrames,
etaSeconds: job.etaSeconds,
errorMessage: job.errorMessage,
machineName: job.machine?.name ?? null,
}
: null,
previewUrl: exp.previewMovKey ? `/api/files/${exp.previewMovKey}` : null,
thumbnailUrl: exp.thumbnailKey ? `/api/files/${exp.thumbnailKey}` : null,
},
};
}
/** E3 — export detail incl. attempts and audit events. */
export async function getExportDetail(exportId: string) {
const exp = await db.export.findUnique({
where: { id: exportId },
include: {
shot: { select: { id: true, shotCode: true, episode: true, project: { select: { id: true, name: true, code: true } } } },
renderJobs: {
orderBy: { attempt: "asc" },
include: { machine: { select: { id: true, name: true } } },
},
events: { orderBy: { createdAt: "asc" } },
},
});
if (!exp) throw new PipelineError(404, "Export not found");
return { export: serializeExport(exp) };
}
/**
* E14 / E15 retry a failed export: new RenderJob attempt, back to QUEUED.
* PREVIEW_FAILED is routed to the preview-only retry (T11) so validated EXRs
* are never re-rendered just to rebuild a slate.
*/
export async function retryExport(exportId: string, actor: TransitionActor) {
const current = await db.export.findUnique({ where: { id: exportId }, select: { status: true } });
if (!current) throw new PipelineError(404, "Export not found");
if (current.status === "PREVIEW_FAILED") {
const { retryPreview } = await import("./preview");
return retryPreview(exportId, actor.id ?? undefined);
}
return db.$transaction(async (tx) => {
const exp = await tx.export.findUnique({
where: { id: exportId },
include: { renderJobs: { orderBy: { attempt: "desc" }, take: 1 } },
});
if (!exp) throw new PipelineError(404, "Export not found");
if (!["RENDER_FAILED", "VALIDATION_FAILED"].includes(exp.status)) {
throw new PipelineError(409, `Export is ${exp.status} — only RENDER_FAILED, VALIDATION_FAILED or PREVIEW_FAILED exports can be retried`);
}
const last = exp.renderJobs[0];
if (!last) throw new PipelineError(500, "Export has no render jobs");
const job = await tx.renderJob.create({
data: {
type: last.type,
exportId: exp.id,
attempt: last.attempt + 1,
maxAttempts: last.maxAttempts,
priority: last.priority,
manifest: last.manifest as Prisma.InputJsonValue,
},
});
const updated = await applyTransition(tx, exp, "QUEUED", {
...actor,
note: actor.note ?? `Retry (attempt ${job.attempt})`,
});
return { export: { id: updated.id, status: updated.status }, renderJob: { id: job.id, attempt: job.attempt } };
});
}
/** E15 / T18 — cancel a queued or rendering export. Worker learns via heartbeat/progress. */
export async function cancelExport(exportId: string, actor: TransitionActor) {
return db.$transaction(async (tx) => {
const exp = await tx.export.findUnique({ where: { id: exportId }, select: { id: true, status: true } });
if (!exp) throw new PipelineError(404, "Export not found");
if (exp.status === "CANCELLED") return { export: exp, alreadyApplied: true };
if (!["QUEUED", "RENDERING"].includes(exp.status)) {
throw new PipelineError(409, `Export is ${exp.status} — only QUEUED or RENDERING exports can be cancelled`);
}
await tx.renderJob.updateMany({
where: { exportId: exp.id, status: "QUEUED" },
data: { status: "CANCELLED", finishedAt: new Date(), errorMessage: "Cancelled by user" },
});
// Claimed/running jobs stay marked CANCELLED but keep finishedAt null until
// the worker acknowledges (heartbeat CANCEL_JOB command / progress cancelRequested)
await tx.renderJob.updateMany({
where: { exportId: exp.id, status: { in: ["CLAIMED", "RUNNING"] } },
data: { status: "CANCELLED", errorMessage: "Cancelled by user" },
});
const updated = await applyTransition(tx, exp, "CANCELLED", actor);
return { export: { id: updated.id, status: updated.status }, alreadyApplied: false };
});
}
/**
* Phase 1 stopgap (§17.1): admin "mark as done manually" so the studio can
* keep using the old render path while the queue is validated. Removed once
* workers render for real.
*/
export async function markExportDoneManually(exportId: string, userId: string, note?: string) {
return db.$transaction(async (tx) => {
const exp = await tx.export.findUnique({ where: { id: exportId }, select: { id: true, status: true } });
if (!exp) throw new PipelineError(404, "Export not found");
if (!["QUEUED", "RENDERING"].includes(exp.status)) {
throw new PipelineError(409, `Export is ${exp.status} — only QUEUED or RENDERING exports can be marked done`);
}
await tx.renderJob.updateMany({
where: { exportId: exp.id, status: { in: [...OPEN_JOB_STATUSES] } },
data: { status: "CANCELLED", finishedAt: new Date(), errorMessage: "Manually marked done (legacy render path)" },
});
const updated = await applyTransition(tx, exp, "READY_FOR_QC", {
type: "USER",
id: userId,
note: note ?? "Manually marked done (rendered outside the pipeline)",
});
return { export: { id: updated.id, status: updated.status } };
});
}
export interface ListExportsFilter {
projectId?: string;
episode?: string;
status?: ExportStatus;
shotId?: string;
page?: number;
limit?: number;
}
/** E21 — paginated export list for the web queue/monitoring pages. */
export async function listExports(filter: ListExportsFilter) {
const page = Math.max(1, filter.page ?? 1);
const limit = Math.min(200, Math.max(1, filter.limit ?? 50));
const where: Prisma.ExportWhereInput = {
...(filter.projectId ? { projectId: filter.projectId } : {}),
...(filter.status ? { status: filter.status } : {}),
...(filter.shotId ? { shotId: filter.shotId } : {}),
...(filter.episode ? { shot: { episode: filter.episode } } : {}),
};
const [total, rows] = await Promise.all([
db.export.count({ where }),
db.export.findMany({
where,
orderBy: { createdAt: "desc" },
skip: (page - 1) * limit,
take: limit,
include: {
shot: { select: { shotCode: true, episode: true, project: { select: { name: true, code: true } } } },
renderJobs: {
orderBy: { attempt: "desc" },
take: 1,
include: { machine: { select: { name: true } } },
},
},
}),
]);
return {
pagination: { page, limit, total, pages: Math.ceil(total / limit) },
exports: rows.map((e) => ({
id: e.id,
shotCode: e.shot.shotCode,
episode: e.shot.episode,
projectName: e.shot.project.name,
projectCode: e.shot.project.code,
versionString: e.versionString,
status: e.status,
statusChangedAt: e.statusChangedAt,
createdAt: e.createdAt,
outputDir: e.outputDir,
job: e.renderJobs[0]
? {
id: e.renderJobs[0].id,
attempt: e.renderJobs[0].attempt,
status: e.renderJobs[0].status,
progress: e.renderJobs[0].progress,
currentFrame: e.renderJobs[0].currentFrame,
totalFrames: e.renderJobs[0].totalFrames,
etaSeconds: e.renderJobs[0].etaSeconds,
priority: e.renderJobs[0].priority,
machineName: e.renderJobs[0].machine?.name ?? null,
errorMessage: e.renderJobs[0].errorMessage,
}
: null,
})),
};
}
+14
View File
@@ -0,0 +1,14 @@
import { NextRequest } from "next/server";
/**
* Shared API-key auth for /api/ext/* pipeline routes same contract as the
* existing ext endpoints (Authorization: Bearer <key> or X-Api-Key header).
* One shared API_SECRET_KEY for now (studio decision 18.1-Q7).
*/
export function isExtAuthorized(req: NextRequest): boolean {
const apiKey = process.env.API_SECRET_KEY;
if (!apiKey) return false;
const authHeader = req.headers.get("authorization") ?? "";
if (authHeader.startsWith("Bearer ")) return authHeader.slice(7) === apiKey;
return (req.headers.get("x-api-key") ?? "") === apiKey;
}
+317
View File
@@ -0,0 +1,317 @@
import { Prisma, RenderJobType } from "@prisma/client";
import { db } from "@/lib/db";
import { PipelineError } from "./errors";
import { applyTransition } from "./transitions";
import { getConfigBoolean, getConfigNumber } from "./config";
import { machineCanClaim } from "./machines";
import { reapExpiredLeases } from "./reaper";
import { buildPreviewManifest, createPreviewJob } from "./preview";
const JOB_TYPES: RenderJobType[] = ["AE_RENDER", "PREVIEW_ONLY", "DELIVERY_BUILD"];
/**
* E8 atomically claim the next queued job (§6.4). `FOR UPDATE SKIP LOCKED`
* makes double-claims impossible under concurrent workers. All scheduling
* policy (availability windows, Render Now, urgent priority) is enforced
* here server-side workers poll dumbly (§7.10).
*/
export async function claimNextJob(machineId: string, types?: string[]) {
const machine = await db.machine.findUnique({ where: { id: machineId } });
if (!machine) throw new PipelineError(404, "Machine not found — register first (POST /api/ext/workers/register)");
// Defensive sweep on each claim (studio decision 18.1-Q8)
await reapExpiredLeases().catch((err) => console.error("[render-pipeline] claim-time reap failed", err));
const claimTypes = (types?.length ? types : ["AE_RENDER", "PREVIEW_ONLY", "DELIVERY_BUILD"]).filter(
(t): t is RenderJobType => (JOB_TYPES as string[]).includes(t)
);
if (claimTypes.length === 0) throw new PipelineError(422, "No valid job types requested");
const gate = machineCanClaim(machine, new Date());
if (gate === "none") return null;
const [leaseSeconds, urgentThreshold] = await Promise.all([
getConfigNumber("render.leaseSeconds"),
getConfigNumber("render.urgentPriorityThreshold"),
]);
const urgentFilter =
gate === "urgent-only" ? Prisma.sql`AND priority <= ${urgentThreshold}` : Prisma.empty;
const claimed = await db.$queryRaw<{ id: string }[]>(Prisma.sql`
UPDATE "render_jobs"
SET status = 'CLAIMED',
"machineId" = ${machineId},
"claimedAt" = now(),
"leaseExpiresAt" = now() + (${leaseSeconds}::int * interval '1 second'),
"updatedAt" = now()
WHERE id = (
SELECT id FROM "render_jobs"
WHERE status = 'QUEUED'
AND type::text = ANY(${claimTypes}::text[])
${urgentFilter}
ORDER BY priority ASC, "createdAt" ASC
FOR UPDATE SKIP LOCKED
LIMIT 1
)
RETURNING id
`);
if (claimed.length === 0) return null;
const job = await db.renderJob.findUniqueOrThrow({
where: { id: claimed[0].id },
include: { export: { select: { id: true, status: true, versionString: true } } },
});
// T2: QUEUED → RENDERING
if (job.export && job.export.status === "QUEUED") {
await db.$transaction(async (tx) => {
await applyTransition(tx, job.export!, "RENDERING", {
type: "WORKER",
id: machineId,
note: `Claimed by ${machine.name} (attempt ${job.attempt})`,
});
});
}
return {
job: {
id: job.id,
type: job.type,
attempt: job.attempt,
maxAttempts: job.maxAttempts,
priority: job.priority,
exportId: job.exportId,
deliveryId: job.deliveryId,
leaseExpiresAt: job.leaseExpiresAt,
manifest: job.manifest,
},
};
}
async function getJobForReport(jobId: string, machineId: string) {
const job = await db.renderJob.findUnique({
where: { id: jobId },
include: { export: { select: { id: true, status: true } } },
});
if (!job) throw new PipelineError(404, "Render job not found");
// Zombie-worker guard (§7.4): reports from a machine that doesn't hold the lease are rejected
if (job.machineId !== machineId) {
throw new PipelineError(409, "Job is not held by this machine (lease reassigned?)");
}
return job;
}
/** E9 — progress report; renews the lease. */
export async function reportProgress(
jobId: string,
machineId: string,
body: { progress?: number; currentFrame?: number; totalFrames?: number; etaSeconds?: number; logTail?: string }
) {
const job = await getJobForReport(jobId, machineId);
if (job.status === "CANCELLED") {
return { ok: true, cancelRequested: true, leaseExpiresAt: job.leaseExpiresAt };
}
if (["COMPLETED", "FAILED", "EXPIRED"].includes(job.status)) {
return { ok: true, alreadyApplied: true, cancelRequested: false, leaseExpiresAt: job.leaseExpiresAt };
}
const leaseSeconds = await getConfigNumber("render.leaseSeconds");
const leaseExpiresAt = new Date(Date.now() + leaseSeconds * 1000);
await db.renderJob.update({
where: { id: job.id },
data: {
status: "RUNNING",
startedAt: job.startedAt ?? new Date(),
leaseExpiresAt,
...(body.progress != null ? { progress: Math.min(1, Math.max(0, body.progress)) } : {}),
...(body.currentFrame != null ? { currentFrame: body.currentFrame } : {}),
...(body.totalFrames != null ? { totalFrames: body.totalFrames } : {}),
...(body.etaSeconds != null ? { etaSeconds: body.etaSeconds } : {}),
...(body.logTail != null ? { logTail: body.logTail.slice(-20000) } : {}),
},
});
return { ok: true, leaseExpiresAt, cancelRequested: false };
}
/** E10 — failure report. Auto-requeues while attempts remain and the error is retryable. */
export async function reportFail(
jobId: string,
machineId: string,
body: {
stage?: string;
exitCode?: number;
errorMessage?: string;
logTail?: string;
logFileKey?: string;
retryable?: boolean;
}
) {
return db.$transaction(async (tx) => {
const job = await tx.renderJob.findUnique({
where: { id: jobId },
include: { export: { select: { id: true, status: true } } },
});
if (!job) throw new PipelineError(404, "Render job not found");
if (job.machineId !== machineId) throw new PipelineError(409, "Job is not held by this machine (lease reassigned?)");
if (job.status === "FAILED") {
return { job: { id: job.id, status: job.status }, export: job.export ? { id: job.export.id, status: job.export.status } : null, autoRequeued: false, alreadyApplied: true };
}
if (job.status === "CANCELLED") {
// worker acknowledging a cancel — record the wind-down, stay CANCELLED
await tx.renderJob.update({ where: { id: job.id }, data: { finishedAt: job.finishedAt ?? new Date(), logTail: body.logTail?.slice(-20000) ?? job.logTail } });
return { job: { id: job.id, status: "CANCELLED" }, export: job.export ? { id: job.export.id, status: job.export.status } : null, autoRequeued: false, alreadyApplied: true };
}
if (job.status === "COMPLETED") {
throw new PipelineError(409, "Job already reported complete — contradictory fail report rejected");
}
await tx.renderJob.update({
where: { id: job.id },
data: {
status: "FAILED",
finishedAt: new Date(),
exitCode: body.exitCode ?? null,
errorMessage: body.errorMessage?.slice(0, 2000) ?? null,
logTail: body.logTail?.slice(-20000) ?? job.logTail,
logFileKey: body.logFileKey ?? job.logFileKey,
},
});
if (!job.export) {
return { job: { id: job.id, status: "FAILED" }, export: null, autoRequeued: false };
}
const retryable = body.retryable !== false;
const autoRequeue = retryable && job.attempt < job.maxAttempts;
const isPreview = job.type === "PREVIEW_ONLY";
if (autoRequeue) {
const next = await tx.renderJob.create({
data: {
type: job.type,
exportId: job.exportId,
attempt: job.attempt + 1,
maxAttempts: job.maxAttempts,
priority: job.priority,
manifest: job.manifest as Prisma.InputJsonValue,
},
});
const note = `${body.stage ?? "RENDER"} failed (${body.errorMessage ?? "no message"}) — auto-requeued as attempt ${next.attempt}`;
if (isPreview) {
// The export stays in GENERATING_PREVIEW across preview attempts —
// the EXRs are untouched, only the preview build is repeated.
await tx.exportEvent.create({
data: {
exportId: job.export.id,
fromStatus: job.export.status,
toStatus: job.export.status,
actorType: "SYSTEM",
actorId: machineId,
note,
},
});
return { job: { id: job.id, status: "FAILED" }, export: { id: job.export.id, status: job.export.status }, autoRequeued: true, nextJob: { id: next.id, attempt: next.attempt } };
}
const exp = await applyTransition(tx, job.export, "QUEUED", { type: "SYSTEM", id: machineId, note });
return { job: { id: job.id, status: "FAILED" }, export: { id: exp.id, status: exp.status }, autoRequeued: true, nextJob: { id: next.id, attempt: next.attempt } };
}
const exp = await applyTransition(tx, job.export, isPreview ? "PREVIEW_FAILED" : "RENDER_FAILED", {
type: "WORKER",
id: machineId,
note: `${body.stage ?? "RENDER"} failed: ${body.errorMessage ?? "no message"}${retryable ? " (max attempts reached)" : " (not retryable)"}`,
});
return { job: { id: job.id, status: "FAILED" }, export: { id: exp.id, status: exp.status }, autoRequeued: false };
});
}
/**
* E11 aerender finished OK.
* Phase 2 interim (§17.3): transitions straight to a provisional READY_FOR_QC;
* Phase 3 will insert VALIDATING between (see transitions.ts note).
*/
export async function completeRender(
jobId: string,
machineId: string,
body: { renderSeconds?: number; logFileKey?: string; logTail?: string; exrFileCount?: number; exrTotalBytes?: number }
) {
return db.$transaction(async (tx) => {
const job = await tx.renderJob.findUnique({
where: { id: jobId },
include: { export: { select: { id: true, status: true } } },
});
if (!job) throw new PipelineError(404, "Render job not found");
if (job.machineId !== machineId) throw new PipelineError(409, "Job is not held by this machine (lease reassigned?)");
if (job.status === "COMPLETED") {
return { job: { id: job.id, status: job.status }, export: job.export ? { id: job.export.id, status: job.export.status } : null, alreadyApplied: true };
}
if (["FAILED", "CANCELLED", "EXPIRED"].includes(job.status)) {
throw new PipelineError(409, `Job is ${job.status} — contradictory complete report rejected`);
}
await tx.renderJob.update({
where: { id: job.id },
data: {
status: "COMPLETED",
progress: 1,
finishedAt: new Date(),
exitCode: 0,
logFileKey: body.logFileKey ?? job.logFileKey,
logTail: body.logTail?.slice(-20000) ?? job.logTail,
},
});
let exportResult: { id: string; status: string } | null = null;
let previewJob: { id: string; type: string } | null = null;
if (job.export) {
const exportRow = await tx.export.findUniqueOrThrow({ where: { id: job.export.id } });
const stats = {
...(body.exrFileCount != null ? { exrFileCount: body.exrFileCount } : {}),
...(body.exrTotalBytes != null ? { exrTotalBytes: BigInt(Math.round(body.exrTotalBytes)) } : {}),
};
// The EXR render is only the first stage: hand off to the preview stage
// (headless AE rebuild → delivery MOV + review MP4) when it is enabled.
const previewEnabled = await getConfigBoolean("preview.enabled");
if (previewEnabled) {
const manifest = await buildPreviewManifest(exportRow);
const created = await createPreviewJob(tx, exportRow, manifest, job.maxAttempts, job.priority);
previewJob = { id: created.id, type: created.type };
const exp = await applyTransition(tx, job.export, "GENERATING_PREVIEW", {
type: "WORKER",
id: machineId,
note: `Render complete in ${body.renderSeconds ?? "?"}s — queued preview build`,
}, stats);
exportResult = { id: exp.id, status: exp.status };
} else {
const exp = await applyTransition(tx, job.export, "READY_FOR_QC", {
type: "WORKER",
id: machineId,
note: `Render complete in ${body.renderSeconds ?? "?"}s (preview stage disabled)`,
}, stats);
exportResult = { id: exp.id, status: exp.status };
}
}
return { job: { id: job.id, status: "COMPLETED" }, export: exportResult, previewJob, alreadyApplied: false };
});
}
/** E15 — cancel a single job from the web UI (delegates to export-level cancel when linked). */
export async function getJobDetail(jobId: string) {
const job = await db.renderJob.findUnique({
where: { id: jobId },
include: {
machine: { select: { id: true, name: true } },
export: { include: { shot: { select: { shotCode: true } } } },
},
});
if (!job) throw new PipelineError(404, "Render job not found");
return job;
}
+240
View File
@@ -0,0 +1,240 @@
import { Machine, Prisma } from "@prisma/client";
import { db } from "@/lib/db";
import { PipelineError } from "./errors";
import { getWorkerConfig } from "./config";
/**
* Machine availability (§7.10):
* ALWAYS claims whenever idle (default when availability is null)
* SCHEDULE claims only inside configured windows
* MANUAL never claims unless explicitly triggered
* Overrides, in priority order: enabled=false beats everything; renderNowUntil
* allows normal claims until it expires; urgent jobs may claim anytime on
* machines with allowUrgentAnytime.
*/
export interface AvailabilityWindow {
days: string[]; // ["mon", ..., "sun"]
from: string; // "19:00"
to: string; // "08:00" — from > to spans midnight
}
export interface MachineAvailability {
mode?: "ALWAYS" | "SCHEDULE" | "MANUAL";
windows?: AvailabilityWindow[];
allowUrgentAnytime?: boolean;
}
const DAY_NAMES = ["sun", "mon", "tue", "wed", "thu", "fri", "sat"];
function parseHm(hm: string): number | null {
const m = /^(\d{1,2}):(\d{2})$/.exec(hm);
if (!m) return null;
const mins = parseInt(m[1], 10) * 60 + parseInt(m[2], 10);
return mins >= 0 && mins <= 24 * 60 ? mins : null;
}
export function isInWindow(windows: AvailabilityWindow[], now: Date): boolean {
const day = DAY_NAMES[now.getDay()];
const prevDay = DAY_NAMES[(now.getDay() + 6) % 7];
const t = now.getHours() * 60 + now.getMinutes();
for (const w of windows) {
const from = parseHm(w.from);
const to = parseHm(w.to);
if (from == null || to == null || !Array.isArray(w.days)) continue;
const days = w.days.map((d) => d.toLowerCase().slice(0, 3));
if (from < to) {
if (days.includes(day) && t >= from && t < to) return true;
} else if (from > to) {
// overnight window, e.g. 19:00 → 08:00
if (days.includes(day) && t >= from) return true;
if (days.includes(prevDay) && t < to) return true;
} else {
// from === to → full 24 h on listed days
if (days.includes(day)) return true;
}
}
return false;
}
export type ClaimGate = "normal" | "urgent-only" | "none";
export function machineCanClaim(machine: Machine, now: Date): ClaimGate {
if (!machine.enabled) return "none";
if (machine.renderNowUntil && machine.renderNowUntil > now) return "normal";
const availability = (machine.availability ?? null) as MachineAvailability | null;
const mode = availability?.mode ?? "ALWAYS";
const urgent = availability?.allowUrgentAnytime ? ("urgent-only" as const) : ("none" as const);
switch (mode) {
case "ALWAYS":
return "normal";
case "SCHEDULE":
return isInWindow(availability?.windows ?? [], now) ? "normal" : urgent;
case "MANUAL":
return urgent;
default:
return "normal";
}
}
/** E6 — idempotent register/upsert on machine name. */
export async function registerMachine(body: {
name?: string;
hostname?: string;
workerVersion?: string;
aeVersion?: string;
capabilities?: unknown;
}) {
const name = body.name?.trim();
if (!name) throw new PipelineError(422, "name is required");
const hostname = body.hostname?.trim() || name;
const machine = await db.machine.upsert({
where: { name },
create: {
name,
hostname,
status: "ONLINE",
lastSeenAt: new Date(),
workerVersion: body.workerVersion ?? null,
aeVersion: body.aeVersion ?? null,
capabilities: (body.capabilities ?? undefined) as Prisma.InputJsonValue | undefined,
},
update: {
hostname,
status: "ONLINE",
lastSeenAt: new Date(),
workerVersion: body.workerVersion ?? undefined,
aeVersion: body.aeVersion ?? undefined,
capabilities: (body.capabilities ?? undefined) as Prisma.InputJsonValue | undefined,
},
});
return {
machine: { id: machine.id, name: machine.name, enabled: machine.enabled },
config: await getWorkerConfig(),
};
}
/** E7 — heartbeat; also the server→worker command channel (cancel signals). */
export async function recordHeartbeat(
machineId: string,
body: { cpuPercent?: number; memPercent?: number; diskFreeGb?: number; currentJobId?: string | null }
) {
const machine = await db.machine.findUnique({ where: { id: machineId } });
if (!machine) throw new PipelineError(404, "Machine not found");
await db.$transaction([
db.machine.update({
where: { id: machineId },
data: { lastSeenAt: new Date(), status: machine.enabled ? "ONLINE" : "DISABLED" },
}),
db.workerHeartbeat.create({
data: {
machineId,
cpuPercent: body.cpuPercent ?? null,
memPercent: body.memPercent ?? null,
diskFreeGb: body.diskFreeGb ?? null,
currentJobId: body.currentJobId ?? null,
},
}),
]);
// Cancelled-but-unacknowledged jobs held by this machine → CANCEL_JOB commands
const cancelled = await db.renderJob.findMany({
where: { machineId, status: "CANCELLED", finishedAt: null },
select: { id: true },
});
return {
ok: true,
commands: cancelled.map((j) => ({ type: "CANCEL_JOB" as const, jobId: j.id })),
};
}
/** Machine list for the monitoring page, with latest heartbeat + current job. */
export async function listMachines() {
const heartbeatSeconds = 30;
const machines = await db.machine.findMany({
orderBy: { name: "asc" },
include: {
heartbeats: { orderBy: { createdAt: "desc" }, take: 1 },
renderJobs: {
where: { status: { in: ["CLAIMED", "RUNNING"] } },
take: 1,
include: { export: { select: { id: true, versionString: true, shot: { select: { shotCode: true } } } } },
},
},
});
const now = Date.now();
return machines.map((m) => ({
id: m.id,
name: m.name,
hostname: m.hostname,
enabled: m.enabled,
// ONLINE = heartbeat within 3× the heartbeat interval (§5.6)
status: !m.enabled
? "DISABLED"
: m.lastSeenAt && now - m.lastSeenAt.getTime() < heartbeatSeconds * 3 * 1000
? "ONLINE"
: "OFFLINE",
lastSeenAt: m.lastSeenAt,
workerVersion: m.workerVersion,
aeVersion: m.aeVersion,
capabilities: m.capabilities,
availability: m.availability,
renderNowUntil: m.renderNowUntil,
latestHeartbeat: m.heartbeats[0]
? {
createdAt: m.heartbeats[0].createdAt,
cpuPercent: m.heartbeats[0].cpuPercent,
memPercent: m.heartbeats[0].memPercent,
diskFreeGb: m.heartbeats[0].diskFreeGb,
}
: null,
currentJob: m.renderJobs[0]
? {
id: m.renderJobs[0].id,
exportId: m.renderJobs[0].exportId,
shotCode: m.renderJobs[0].export?.shot.shotCode ?? null,
versionString: m.renderJobs[0].export?.versionString ?? null,
progress: m.renderJobs[0].progress,
etaSeconds: m.renderJobs[0].etaSeconds,
}
: null,
}));
}
export async function updateMachine(
machineId: string,
patch: { enabled?: boolean; availability?: MachineAvailability | null; renderNowHours?: number | null }
) {
const machine = await db.machine.findUnique({ where: { id: machineId } });
if (!machine) throw new PipelineError(404, "Machine not found");
const data: Prisma.MachineUpdateInput = {};
if (patch.enabled !== undefined) {
data.enabled = patch.enabled;
data.status = patch.enabled ? machine.status === "DISABLED" ? "OFFLINE" : machine.status : "DISABLED";
}
if (patch.availability !== undefined) {
data.availability = patch.availability === null ? Prisma.DbNull : (patch.availability as unknown as Prisma.InputJsonValue);
}
if (patch.renderNowHours !== undefined) {
data.renderNowUntil =
patch.renderNowHours === null || patch.renderNowHours <= 0
? null
: new Date(Date.now() + patch.renderNowHours * 3600 * 1000);
}
const updated = await db.machine.update({ where: { id: machineId }, data });
return {
machine: {
id: updated.id,
name: updated.name,
enabled: updated.enabled,
availability: updated.availability,
renderNowUntil: updated.renderNowUntil,
},
};
}
+52
View File
@@ -0,0 +1,52 @@
import { z } from "zod";
/**
* The Render Manifest (RenderPipeline2 §6.2).
* Uploaded by the AE panel on Queue Export, snapshotted verbatim into
* RenderJob.manifest, handed to the worker on claim.
*
* `outputDir` / `outputPattern` may be the literal string "auto" the server
* generates them from the shot's exrOutput convention and the NEW version
* number it decides (§6.4 E1).
*/
export const renderManifestSchema = z
.object({
manifestVersion: z.number().int().min(1).default(1),
projectCode: z.string().min(1),
shotCode: z.string().regex(/^[A-Za-z0-9_-]+$/, "shotCode contains invalid characters"),
shotId: z.string().min(1),
aepPath: z.string().min(1),
compName: z.string().min(1),
rendererType: z.string().default("aerender"),
aeVersionHint: z.string().optional(),
outputDir: z.string().min(1),
outputPattern: z.string().min(1),
outputModuleTemplate: z.string().optional(),
renderSettingsTemplate: z.string().optional(),
frameStart: z.number().int(),
frameEnd: z.number().int(),
fps: z.number().positive(),
width: z.number().int().positive(),
height: z.number().int().positive(),
expected: z
.object({
colorspace: z.string().optional(),
bitDepth: z.string().optional(),
exrCompression: z.string().optional(),
alpha: z.boolean().optional(),
timecodeStart: z.string().optional(),
})
.optional(),
preview: z
.object({
template: z.string().optional(),
burnins: z.boolean().optional(),
})
.optional(),
})
.refine((m) => m.frameEnd >= m.frameStart, {
message: "frameEnd must be >= frameStart",
path: ["frameEnd"],
});
export type RenderManifest = z.infer<typeof renderManifestSchema>;
+295
View File
@@ -0,0 +1,295 @@
import { Prisma } from "@prisma/client";
import { db } from "@/lib/db";
import { PipelineError } from "./errors";
import { applyTransition } from "./transitions";
import { getConfigNumber, getPreviewConfig } from "./config";
/** Slate date format used by the panel's getDateString (YYYY/MM/DD). */
function slateDate(d = new Date()): string {
const mm = String(d.getMonth() + 1).padStart(2, "0");
const dd = String(d.getDate()).padStart(2, "0");
return `${d.getFullYear()}/${mm}/${dd}`;
}
export interface PreviewManifest {
stage: "PREVIEW";
exportId: string;
shotCode: string;
versionString: string;
outputDir: string;
outputPattern: string;
frameStart: number;
frameEnd: number;
fps: number;
templateAep: string;
templateComp: string;
overlayComp: string;
lutComp: string;
movTemplate: string;
mp4Template: string;
movOutput: string;
mp4Output: string;
slateScopeProp: string;
slateSubmissionProp: string;
slate: {
versionName: string;
date: string;
description: string | null;
notes: string | null;
shotCode: string;
episode: string | null;
scene: string | null;
vfxScope: string | null;
submissionNote: string | null;
};
}
type ExportForPreview = {
id: string;
shotId: string;
versionString: string;
outputDir: string;
outputPattern: string;
frameStart: number;
frameEnd: number;
fps: number;
vfxScope: string | null;
submissionNote: string | null;
};
/**
* Builds the PREVIEW_ONLY job manifest (RenderPipeline2 §9): everything the
* headless AE build script needs to rebuild the shot around the rendered EXRs
* and queue the delivery MOV + review MP4.
*
* The MOV and MP4 are written beside the EXR sequence, so every artifact for a
* version lives in one folder and Phase 5 can package them together.
*/
export async function buildPreviewManifest(exportRow: ExportForPreview): Promise<PreviewManifest> {
const shot = await db.shot.findUnique({
where: { id: exportRow.shotId },
select: { shotCode: true, episode: true, scene: true, description: true, notes: true },
});
if (!shot) throw new PipelineError(404, "Shot not found for export");
const cfg = await getPreviewConfig();
const base = `${shot.shotCode}_cmp_TT_${exportRow.versionString}`;
return {
stage: "PREVIEW",
exportId: exportRow.id,
shotCode: shot.shotCode,
versionString: exportRow.versionString,
outputDir: exportRow.outputDir,
outputPattern: exportRow.outputPattern,
frameStart: exportRow.frameStart,
frameEnd: exportRow.frameEnd,
fps: exportRow.fps,
templateAep: cfg.templateAep,
templateComp: cfg.templateComp,
overlayComp: cfg.overlayComp,
lutComp: cfg.lutComp,
movTemplate: cfg.movTemplate,
mp4Template: cfg.mp4Template,
movOutput: `${exportRow.outputDir}/${base}.mov`,
mp4Output: `${exportRow.outputDir}/${base}.mp4`,
slateScopeProp: cfg.slateScopeProp,
slateSubmissionProp: cfg.slateSubmissionProp,
slate: {
versionName: base,
date: slateDate(),
description: shot.description,
notes: shot.notes,
shotCode: shot.shotCode,
episode: shot.episode,
scene: shot.scene,
vfxScope: exportRow.vfxScope,
submissionNote: exportRow.submissionNote,
},
};
}
/** Creates the PREVIEW_ONLY job that follows a successful EXR render. */
export async function createPreviewJob(
tx: Prisma.TransactionClient,
exportRow: ExportForPreview,
manifest: PreviewManifest,
maxAttempts: number,
priority: number
) {
return tx.renderJob.create({
data: {
type: "PREVIEW_ONLY",
exportId: exportRow.id,
attempt: 1,
maxAttempts,
priority,
manifest: manifest as unknown as Prisma.InputJsonValue,
},
});
}
export interface FinalizeInput {
artifacts?: {
deliveryMovPath?: string;
previewMovKey?: string;
thumbnailKey?: string;
metadataKey?: string;
logFileKey?: string;
};
renderStats?: { renderSeconds?: number; previewSeconds?: number };
media?: { width?: number; height?: number; frameCount?: number; fps?: number; fileName?: string };
}
/**
* E13 preview artifacts are done: register the review MP4 as an ordinary
* `Version` and move the Export to READY_FOR_QC.
*
* Pipeline mode (§6.4 / T12): the Version is **never client-visible** and
* **no task or shot status is touched**. Delivery QC runs on shots the client
* has usually already approved, so a post-approval technical render must not
* resurface in the review flow or the client portal.
*/
export async function finalizePreview(jobId: string, machineId: string, body: FinalizeInput) {
return db.$transaction(async (tx) => {
const job = await tx.renderJob.findUnique({
where: { id: jobId },
include: { export: true },
});
if (!job) throw new PipelineError(404, "Render job not found");
if (job.machineId !== machineId) {
throw new PipelineError(409, "Job is not held by this machine (lease reassigned?)");
}
const exportRow = job.export;
if (!exportRow) throw new PipelineError(422, "Job has no export to finalize");
if (exportRow.status === "READY_FOR_QC" && exportRow.versionId) {
return {
export: { id: exportRow.id, status: exportRow.status },
version: { id: exportRow.versionId },
alreadyApplied: true,
};
}
const artifacts = body.artifacts ?? {};
const media = body.media ?? {};
await tx.renderJob.update({
where: { id: job.id },
data: {
status: "COMPLETED",
progress: 1,
finishedAt: new Date(),
exitCode: 0,
logFileKey: artifacts.logFileKey ?? job.logFileKey,
},
});
let versionId: string | null = null;
if (artifacts.previewMovKey) {
// Version numbering stays monotonic for the task (existing review flows
// rely on the newest version having the highest number) while matching
// the export version whenever it is ahead.
const taskMax = exportRow.taskId
? await tx.version.aggregate({
where: { taskId: exportRow.taskId },
_max: { versionNumber: true },
})
: { _max: { versionNumber: null as number | null } };
const versionNumber = Math.max((taskMax._max.versionNumber ?? 0) + 1, exportRow.versionNumber);
if (exportRow.taskId) {
await tx.version.updateMany({
where: { taskId: exportRow.taskId },
data: { isLatest: false },
});
}
const version = await tx.version.create({
data: {
versionNumber,
shotId: exportRow.shotId,
taskId: exportRow.taskId,
artistId: exportRow.submittedById,
fileUrl: `/api/files/${artifacts.previewMovKey}`,
fileName: media.fileName ?? `${exportRow.versionString}.mp4`,
mimeType: "video/mp4",
thumbnailUrl: artifacts.thumbnailKey ? `/api/files/${artifacts.thumbnailKey}` : undefined,
fps: media.fps ?? exportRow.fps,
frameCount: media.frameCount ?? exportRow.frameEnd - exportRow.frameStart + 1,
width: media.width ?? exportRow.width,
height: media.height ?? exportRow.height,
notes: `Pipeline render ${exportRow.versionString} (internal only)`,
isLatest: true,
isClientVisible: false, // never shared to the client portal (§10.0)
},
});
versionId = version.id;
}
const updated = await applyTransition(
tx,
exportRow,
"READY_FOR_QC",
{
type: "WORKER",
id: machineId,
note: `Preview complete in ${body.renderStats?.previewSeconds ?? "?"}s${versionId ? " — review version created" : " — no preview media reported"}`,
},
{
...(versionId ? { versionId } : {}),
...(artifacts.deliveryMovPath ? { deliveryMovPath: artifacts.deliveryMovPath } : {}),
...(artifacts.previewMovKey ? { previewMovKey: artifacts.previewMovKey } : {}),
...(artifacts.thumbnailKey ? { thumbnailKey: artifacts.thumbnailKey } : {}),
...(artifacts.metadataKey ? { metadataKey: artifacts.metadataKey } : {}),
}
);
return {
export: { id: updated.id, status: updated.status },
version: versionId ? { id: versionId } : null,
alreadyApplied: false,
};
});
}
/** T11 — retry the preview stage only; validated EXRs are left alone. */
export async function retryPreview(exportId: string, actorId?: string) {
const maxAttempts = await getConfigNumber("render.maxAttempts");
return db.$transaction(async (tx) => {
const exportRow = await tx.export.findUnique({ where: { id: exportId } });
if (!exportRow) throw new PipelineError(404, "Export not found");
if (exportRow.status !== "PREVIEW_FAILED") {
throw new PipelineError(409, `Export is ${exportRow.status} — only PREVIEW_FAILED exports can retry the preview`);
}
const manifest = await buildPreviewManifest(exportRow);
const last = await tx.renderJob.findFirst({
where: { exportId, type: "PREVIEW_ONLY" },
orderBy: { attempt: "desc" },
});
const job = await tx.renderJob.create({
data: {
type: "PREVIEW_ONLY",
exportId,
attempt: (last?.attempt ?? 0) + 1,
maxAttempts,
priority: last?.priority ?? 50,
manifest: manifest as unknown as Prisma.InputJsonValue,
},
});
const updated = await applyTransition(tx, exportRow, "GENERATING_PREVIEW", {
type: "USER",
id: actorId,
note: `Retry preview (attempt ${job.attempt})`,
});
return {
export: { id: updated.id, status: updated.status },
renderJob: { id: job.id, attempt: job.attempt },
};
});
}
+94
View File
@@ -0,0 +1,94 @@
import { Prisma } from "@prisma/client";
import { db } from "@/lib/db";
import { applyTransition } from "./transitions";
import { getConfigNumber } from "./config";
/**
* Server-side reaper (RenderPipeline2 §7.7) the only background job the web
* app gains. Covers worker power loss, BSOD and network partition with no
* worker cooperation:
* 1. expired leases job EXPIRED; requeue (T5) or Export RENDER_FAILED (T4)
* 2. silent machines OFFLINE
* 3. prune heartbeats older than 7 days
*/
export async function reapExpiredLeases() {
const expired = await db.renderJob.findMany({
where: { status: { in: ["CLAIMED", "RUNNING"] }, leaseExpiresAt: { lt: new Date() } },
include: { export: { select: { id: true, status: true } } },
});
for (const job of expired) {
try {
await db.$transaction(async (tx) => {
// Re-check inside the transaction — the worker may have reported in between
const fresh = await tx.renderJob.findUnique({ where: { id: job.id }, select: { status: true, leaseExpiresAt: true } });
if (!fresh || !["CLAIMED", "RUNNING"].includes(fresh.status) || !fresh.leaseExpiresAt || fresh.leaseExpiresAt >= new Date()) {
return;
}
await tx.renderJob.update({
where: { id: job.id },
data: { status: "EXPIRED", finishedAt: new Date(), errorMessage: "Lease expired (worker unreachable)" },
});
if (!job.export) return;
if (job.attempt < job.maxAttempts) {
const next = await tx.renderJob.create({
data: {
type: job.type,
exportId: job.exportId,
attempt: job.attempt + 1,
maxAttempts: job.maxAttempts,
priority: job.priority,
manifest: job.manifest as Prisma.InputJsonValue,
},
});
if (job.export.status === "RENDERING") {
await applyTransition(tx, job.export, "QUEUED", {
type: "SYSTEM",
note: `Lease expired on attempt ${job.attempt} — requeued as attempt ${next.attempt}`,
});
}
} else if (job.export.status === "RENDERING") {
await applyTransition(tx, job.export, "RENDER_FAILED", {
type: "SYSTEM",
note: `Lease expired on attempt ${job.attempt}/${job.maxAttempts} — no retries remain`,
});
}
});
} catch (err) {
console.error(`[render-pipeline] reaper failed for job ${job.id}`, err);
}
}
return expired.length;
}
export async function reaperSweep() {
const heartbeatSeconds = await getConfigNumber("render.heartbeatSeconds");
await reapExpiredLeases();
await db.machine.updateMany({
where: {
status: "ONLINE",
OR: [{ lastSeenAt: null }, { lastSeenAt: { lt: new Date(Date.now() - heartbeatSeconds * 3 * 1000) } }],
},
data: { status: "OFFLINE" },
});
await db.workerHeartbeat.deleteMany({
where: { createdAt: { lt: new Date(Date.now() - 7 * 24 * 3600 * 1000) } },
});
}
const globalForReaper = globalThis as unknown as { renderPipelineReaper?: ReturnType<typeof setInterval> };
/** Started once from instrumentation.ts (Next.js server startup hook). */
export function startReaper(intervalMs = 60_000) {
if (globalForReaper.renderPipelineReaper) return;
globalForReaper.renderPipelineReaper = setInterval(() => {
reaperSweep().catch((err) => console.error("[render-pipeline] reaper sweep failed", err));
}, intervalMs);
// Don't hold the process open on shutdown
globalForReaper.renderPipelineReaper.unref?.();
console.log("[render-pipeline] reaper started (interval", intervalMs, "ms)");
}
+22
View File
@@ -0,0 +1,22 @@
import { auth } from "@/auth";
import { PipelineError } from "./errors";
export type SessionUser = { id: string; role: string; name?: string | null; email?: string | null };
/** Any signed-in studio user (clients never see pipeline pages). */
export async function requirePipelineUser(): Promise<SessionUser> {
const session = await auth();
const user = session?.user as (SessionUser & { role?: string }) | undefined;
if (!user?.id) throw new PipelineError(401, "Unauthorized");
if (user.role === "CLIENT") throw new PipelineError(403, "Forbidden");
return user as SessionUser;
}
/** Admin-level pipeline actions (machine kill-switch, manual mark-done). */
export async function requirePipelineAdmin(): Promise<SessionUser> {
const user = await requirePipelineUser();
if (!["ADMIN", "PRODUCER", "SUPERVISOR"].includes(user.role)) {
throw new PipelineError(403, "Forbidden — requires admin/producer/supervisor role");
}
return user;
}
+77
View File
@@ -0,0 +1,77 @@
import { ExportStatus, Prisma } from "@prisma/client";
import { PipelineError } from "./errors";
/**
* Export lifecycle state machine (RenderPipeline2 §4).
* The server is the sole authority on state; workers *request* transitions.
*
* NOTE (Phase 2 interim): RENDERING READY_FOR_QC is temporarily legal.
* Phase 3 inserts VALIDATING between them (§17.3) remove READY_FOR_QC from
* RENDERING's list when the validation engine ships.
*/
export const EXPORT_TRANSITIONS: Record<ExportStatus, ExportStatus[]> = {
QUEUED: ["RENDERING", "CANCELLED", "SUPERSEDED", "READY_FOR_QC"], // READY_FOR_QC: manual mark-done only
RENDERING: [
"VALIDATING",
"GENERATING_PREVIEW", // interim: preview stage runs before validation exists
"RENDER_FAILED",
"QUEUED", // lease expiry / retryable fail with attempts remaining (T5)
"CANCELLED",
"READY_FOR_QC", // preview disabled / manual mark-done
],
RENDER_FAILED: ["QUEUED", "SUPERSEDED"],
VALIDATING: ["VALIDATION_FAILED", "GENERATING_PREVIEW"],
VALIDATION_FAILED: ["QUEUED", "SUPERSEDED"],
GENERATING_PREVIEW: ["PREVIEW_FAILED", "READY_FOR_QC"],
PREVIEW_FAILED: ["GENERATING_PREVIEW", "SUPERSEDED"],
READY_FOR_QC: ["QC_FAILED", "READY_FOR_DELIVERY", "SUPERSEDED"],
QC_FAILED: ["SUPERSEDED"],
READY_FOR_DELIVERY: ["PACKAGED", "SUPERSEDED"],
PACKAGED: ["DELIVERED"],
DELIVERED: ["ARCHIVED"],
SUPERSEDED: [],
ARCHIVED: [],
CANCELLED: [],
};
export type TransitionActor = {
type: "WORKER" | "USER" | "SYSTEM";
id?: string | null;
note?: string | null;
};
export function isTransitionAllowed(from: ExportStatus, to: ExportStatus): boolean {
return EXPORT_TRANSITIONS[from]?.includes(to) ?? false;
}
/**
* Applies a state transition inside an existing transaction: validates
* legality, stamps statusChangedAt, appends the ExportEvent audit row.
* Throws PipelineError(409) on an illegal transition (§4.2 rules).
*/
export async function applyTransition(
tx: Prisma.TransactionClient,
exportRow: { id: string; status: ExportStatus },
to: ExportStatus,
actor: TransitionActor,
extraData: Prisma.ExportUpdateInput = {}
) {
if (!isTransitionAllowed(exportRow.status, to)) {
throw new PipelineError(409, `Invalid transition ${exportRow.status}${to}`);
}
const updated = await tx.export.update({
where: { id: exportRow.id },
data: { status: to, statusChangedAt: new Date(), ...extraData },
});
await tx.exportEvent.create({
data: {
exportId: exportRow.id,
fromStatus: exportRow.status,
toStatus: to,
actorType: actor.type,
actorId: actor.id ?? null,
note: actor.note ?? null,
},
});
return updated;
}
@@ -0,0 +1,178 @@
-- CreateEnum
CREATE TYPE "ExportStatus" AS ENUM ('QUEUED', 'RENDERING', 'RENDER_FAILED', 'VALIDATING', 'VALIDATION_FAILED', 'GENERATING_PREVIEW', 'PREVIEW_FAILED', 'READY_FOR_QC', 'QC_FAILED', 'READY_FOR_DELIVERY', 'PACKAGED', 'DELIVERED', 'SUPERSEDED', 'ARCHIVED', 'CANCELLED');
-- CreateEnum
CREATE TYPE "RenderJobStatus" AS ENUM ('QUEUED', 'CLAIMED', 'RUNNING', 'COMPLETED', 'FAILED', 'CANCELLED', 'EXPIRED');
-- CreateEnum
CREATE TYPE "RenderJobType" AS ENUM ('AE_RENDER', 'PREVIEW_ONLY', 'DELIVERY_BUILD');
-- CreateEnum
CREATE TYPE "ValidationStatus" AS ENUM ('PASS', 'FAIL', 'WARN', 'SKIPPED');
-- CreateEnum
CREATE TYPE "QCResult" AS ENUM ('PASS', 'FAIL');
-- CreateEnum
CREATE TYPE "MachineStatus" AS ENUM ('ONLINE', 'OFFLINE', 'DISABLED');
-- CreateEnum
CREATE TYPE "DeliveryStatus" AS ENUM ('DRAFT', 'QUEUED', 'BUILDING', 'READY', 'DELIVERED', 'FAILED', 'CANCELLED');
-- AlterTable
ALTER TABLE "projects" ADD COLUMN "deliveryConfig" JSONB;
-- CreateTable
CREATE TABLE "exports" (
"id" TEXT NOT NULL,
"shotId" TEXT NOT NULL,
"projectId" TEXT NOT NULL,
"taskId" TEXT,
"versionId" TEXT,
"versionNumber" INTEGER NOT NULL,
"versionString" TEXT NOT NULL,
"status" "ExportStatus" NOT NULL DEFAULT 'QUEUED',
"statusChangedAt" TIMESTAMP(3) NOT NULL DEFAULT CURRENT_TIMESTAMP,
"aepPath" TEXT NOT NULL,
"compName" TEXT NOT NULL,
"rendererType" TEXT NOT NULL,
"outputDir" TEXT NOT NULL,
"outputPattern" TEXT NOT NULL,
"frameStart" INTEGER NOT NULL,
"frameEnd" INTEGER NOT NULL,
"fps" DOUBLE PRECISION NOT NULL,
"width" INTEGER NOT NULL,
"height" INTEGER NOT NULL,
"colorspace" TEXT,
"deliveryMovPath" TEXT,
"previewMovKey" TEXT,
"thumbnailKey" TEXT,
"metadataKey" TEXT,
"exrFileCount" INTEGER,
"exrTotalBytes" BIGINT,
"checksum" TEXT,
"submittedById" TEXT,
"submittedByName" TEXT,
"supersededById" TEXT,
"createdAt" TIMESTAMP(3) NOT NULL DEFAULT CURRENT_TIMESTAMP,
"updatedAt" TIMESTAMP(3) NOT NULL,
CONSTRAINT "exports_pkey" PRIMARY KEY ("id")
);
-- CreateTable
CREATE TABLE "render_jobs" (
"id" TEXT NOT NULL,
"type" "RenderJobType" NOT NULL DEFAULT 'AE_RENDER',
"exportId" TEXT,
"deliveryId" TEXT,
"attempt" INTEGER NOT NULL DEFAULT 1,
"maxAttempts" INTEGER NOT NULL DEFAULT 3,
"status" "RenderJobStatus" NOT NULL DEFAULT 'QUEUED',
"priority" INTEGER NOT NULL DEFAULT 50,
"manifest" JSONB NOT NULL,
"machineId" TEXT,
"claimedAt" TIMESTAMP(3),
"leaseExpiresAt" TIMESTAMP(3),
"startedAt" TIMESTAMP(3),
"finishedAt" TIMESTAMP(3),
"progress" DOUBLE PRECISION NOT NULL DEFAULT 0,
"currentFrame" INTEGER,
"totalFrames" INTEGER,
"etaSeconds" INTEGER,
"exitCode" INTEGER,
"errorMessage" TEXT,
"logTail" TEXT,
"logFileKey" TEXT,
"createdAt" TIMESTAMP(3) NOT NULL DEFAULT CURRENT_TIMESTAMP,
"updatedAt" TIMESTAMP(3) NOT NULL,
CONSTRAINT "render_jobs_pkey" PRIMARY KEY ("id")
);
-- CreateTable
CREATE TABLE "export_events" (
"id" TEXT NOT NULL,
"exportId" TEXT NOT NULL,
"fromStatus" TEXT,
"toStatus" TEXT NOT NULL,
"actorType" TEXT NOT NULL,
"actorId" TEXT,
"note" TEXT,
"createdAt" TIMESTAMP(3) NOT NULL DEFAULT CURRENT_TIMESTAMP,
CONSTRAINT "export_events_pkey" PRIMARY KEY ("id")
);
-- CreateTable
CREATE TABLE "machines" (
"id" TEXT NOT NULL,
"name" TEXT NOT NULL,
"hostname" TEXT NOT NULL,
"status" "MachineStatus" NOT NULL DEFAULT 'OFFLINE',
"enabled" BOOLEAN NOT NULL DEFAULT true,
"lastSeenAt" TIMESTAMP(3),
"workerVersion" TEXT,
"aeVersion" TEXT,
"capabilities" JSONB,
"availability" JSONB,
"renderNowUntil" TIMESTAMP(3),
"apiKeyHash" TEXT,
"createdAt" TIMESTAMP(3) NOT NULL DEFAULT CURRENT_TIMESTAMP,
"updatedAt" TIMESTAMP(3) NOT NULL,
CONSTRAINT "machines_pkey" PRIMARY KEY ("id")
);
-- CreateTable
CREATE TABLE "worker_heartbeats" (
"id" TEXT NOT NULL,
"machineId" TEXT NOT NULL,
"createdAt" TIMESTAMP(3) NOT NULL DEFAULT CURRENT_TIMESTAMP,
"cpuPercent" DOUBLE PRECISION,
"memPercent" DOUBLE PRECISION,
"diskFreeGb" DOUBLE PRECISION,
"currentJobId" TEXT,
CONSTRAINT "worker_heartbeats_pkey" PRIMARY KEY ("id")
);
-- CreateIndex
CREATE INDEX "exports_status_idx" ON "exports"("status");
-- CreateIndex
CREATE INDEX "exports_projectId_status_idx" ON "exports"("projectId", "status");
-- CreateIndex
CREATE UNIQUE INDEX "exports_shotId_versionNumber_key" ON "exports"("shotId", "versionNumber");
-- CreateIndex
CREATE INDEX "render_jobs_status_priority_createdAt_idx" ON "render_jobs"("status", "priority", "createdAt");
-- CreateIndex
CREATE INDEX "render_jobs_machineId_status_idx" ON "render_jobs"("machineId", "status");
-- CreateIndex
CREATE INDEX "export_events_exportId_createdAt_idx" ON "export_events"("exportId", "createdAt");
-- CreateIndex
CREATE UNIQUE INDEX "machines_name_key" ON "machines"("name");
-- CreateIndex
CREATE INDEX "worker_heartbeats_machineId_createdAt_idx" ON "worker_heartbeats"("machineId", "createdAt");
-- AddForeignKey
ALTER TABLE "exports" ADD CONSTRAINT "exports_shotId_fkey" FOREIGN KEY ("shotId") REFERENCES "shots"("id") ON DELETE RESTRICT ON UPDATE CASCADE;
-- AddForeignKey
ALTER TABLE "render_jobs" ADD CONSTRAINT "render_jobs_exportId_fkey" FOREIGN KEY ("exportId") REFERENCES "exports"("id") ON DELETE SET NULL ON UPDATE CASCADE;
-- AddForeignKey
ALTER TABLE "render_jobs" ADD CONSTRAINT "render_jobs_machineId_fkey" FOREIGN KEY ("machineId") REFERENCES "machines"("id") ON DELETE SET NULL ON UPDATE CASCADE;
-- AddForeignKey
ALTER TABLE "export_events" ADD CONSTRAINT "export_events_exportId_fkey" FOREIGN KEY ("exportId") REFERENCES "exports"("id") ON DELETE RESTRICT ON UPDATE CASCADE;
-- AddForeignKey
ALTER TABLE "worker_heartbeats" ADD CONSTRAINT "worker_heartbeats_machineId_fkey" FOREIGN KEY ("machineId") REFERENCES "machines"("id") ON DELETE RESTRICT ON UPDATE CASCADE;
@@ -0,0 +1,5 @@
-- Per-submission slate fields (RenderPipeline2 §9): authored per Export rather
-- than per Shot, because they change from one delivery to the next. New exports
-- default to the previous export's values.
ALTER TABLE "exports" ADD COLUMN "vfxScope" TEXT;
ALTER TABLE "exports" ADD COLUMN "submissionNote" TEXT;
+227
View File
@@ -113,6 +113,72 @@ enum ProjectType {
EPISODIC EPISODIC
} }
// ─────────────────────────────────────────────
// RENDER PIPELINE ENUMS (RenderPipeline2 §5.1)
// ─────────────────────────────────────────────
enum ExportStatus {
QUEUED
RENDERING
RENDER_FAILED
VALIDATING
VALIDATION_FAILED
GENERATING_PREVIEW
PREVIEW_FAILED
READY_FOR_QC
QC_FAILED
READY_FOR_DELIVERY
PACKAGED
DELIVERED
SUPERSEDED
ARCHIVED
CANCELLED
}
enum RenderJobStatus {
QUEUED
CLAIMED
RUNNING
COMPLETED
FAILED
CANCELLED
EXPIRED
}
enum RenderJobType {
AE_RENDER
PREVIEW_ONLY
DELIVERY_BUILD
}
enum ValidationStatus {
PASS
FAIL
WARN
SKIPPED
}
enum QCResult {
PASS
FAIL
}
enum MachineStatus {
ONLINE
OFFLINE
DISABLED
}
enum DeliveryStatus {
DRAFT
QUEUED
BUILDING
READY
DELIVERED
FAILED
CANCELLED
}
enum TakeQuality { enum TakeQuality {
HERO HERO
GOOD GOOD
@@ -292,6 +358,10 @@ model Project {
episodeDueDates EpisodeDueDate[] episodeDueDates EpisodeDueDate[]
shootDays ShootDay[] shootDays ShootDay[]
/// Per-production delivery naming/layout templates (RenderPipeline2 §11.2);
/// falls back to SystemConfig defaults when null
deliveryConfig Json?
@@map("projects") @@map("projects")
} }
@@ -359,6 +429,7 @@ model Shot {
footagePlates FootagePlate[] footagePlates FootagePlate[]
references ShotReference[] references ShotReference[]
loggedTakes Take[] @relation("TakeToShot") loggedTakes Take[] @relation("TakeToShot")
exports Export[]
@@unique([projectId, shotCode]) @@unique([projectId, shotCode])
@@map("shots") @@map("shots")
@@ -752,6 +823,162 @@ model TakeAttachment {
@@map("take_attachments") @@map("take_attachments")
} }
// ─────────────────────────────────────────────
// RENDER PIPELINE MODELS (RenderPipeline2 §5)
// Phases 1 + 2: Export, RenderJob, ExportEvent, Machine, WorkerHeartbeat.
// ValidationResult / QCReview / DeliveryPackage / DeliveryItem arrive in
// later phases as additive migrations.
// ─────────────────────────────────────────────
/// One row per "Queue Export" click — carries the §4 state machine.
model Export {
id String @id @default(cuid())
shotId String
shot Shot @relation(fields: [shotId], references: [id])
projectId String
taskId String? // comp task the preview Version attaches to
versionId String? // preview Version created at READY_FOR_QC (Phase 4)
versionNumber Int // 4
versionString String // "v004"
status ExportStatus @default(QUEUED)
statusChangedAt DateTime @default(now())
// Manifest (denormalised for querying; full manifest JSON on RenderJob)
aepPath String
compName String
rendererType String // "aerender"
outputDir String // render root for this export
outputPattern String // "UNG_106_010_020_cmp_TT_v004.[####].exr"
frameStart Int
frameEnd Int
fps Float
width Int
height Int
colorspace String? // expected, e.g. "ACES - ACEScg"
// Artifacts
deliveryMovPath String? // slate/burn-in delivery MOV on the SAN (§9)
previewMovKey String? // web H.264 transcode in object storage
thumbnailKey String?
metadataKey String? // metadata JSON in object storage
exrFileCount Int?
exrTotalBytes BigInt?
checksum String? // sequence-level digest (xxHash of per-file hashes)
// Slate fields authored per submission (not per shot): they change from one
// delivery to the next, so each Export carries its own and new exports
// default to the previous export's values.
vfxScope String? @db.Text
submissionNote String? @db.Text
submittedById String? // resolved User, else null
submittedByName String? // free text from panel config as fallback
supersededById String? // newer Export that replaced this one
renderJobs RenderJob[]
events ExportEvent[]
createdAt DateTime @default(now())
updatedAt DateTime @updatedAt
@@unique([shotId, versionNumber])
@@index([status])
@@index([projectId, status])
@@map("exports")
}
/// One execution attempt of work by a worker. Also used (type DELIVERY_BUILD)
/// for delivery package builds — one queue/claim/lease/heartbeat mechanism.
model RenderJob {
id String @id @default(cuid())
type RenderJobType @default(AE_RENDER)
exportId String? // set for AE_RENDER / PREVIEW_ONLY
export Export? @relation(fields: [exportId], references: [id])
deliveryId String? // set for DELIVERY_BUILD (Phase 5)
attempt Int @default(1)
maxAttempts Int @default(3)
status RenderJobStatus @default(QUEUED)
priority Int @default(50) // lower = sooner; <= 20 is urgent
manifest Json // full Render Manifest snapshot (§6.2)
machineId String?
machine Machine? @relation(fields: [machineId], references: [id])
claimedAt DateTime?
leaseExpiresAt DateTime? // claim + leaseSeconds; renewed by progress reports
startedAt DateTime?
finishedAt DateTime?
progress Float @default(0) // 0..1
currentFrame Int?
totalFrames Int?
etaSeconds Int?
exitCode Int?
errorMessage String?
logTail String? @db.Text // last ~200 lines of aerender output
logFileKey String? // full log uploaded to object storage on finish/fail
createdAt DateTime @default(now())
updatedAt DateTime @updatedAt
@@index([status, priority, createdAt])
@@index([machineId, status])
@@map("render_jobs")
}
/// Append-only audit log of Export state transitions (§5.9)
model ExportEvent {
id String @id @default(cuid())
exportId String
export Export @relation(fields: [exportId], references: [id])
fromStatus String?
toStatus String
actorType String // "WORKER" | "USER" | "SYSTEM"
actorId String? // machineId or userId
note String?
createdAt DateTime @default(now())
@@index([exportId, createdAt])
@@map("export_events")
}
/// A render-capable machine (artist workstation or future render node) (§5.6)
model Machine {
id String @id @default(cuid())
name String @unique // "RENDER-01"
hostname String
status MachineStatus @default(OFFLINE)
enabled Boolean @default(true) // admin kill-switch: disabled machines cannot claim
lastSeenAt DateTime?
workerVersion String?
aeVersion String? // "2026 (24.x)"
capabilities Json? // { maxConcurrentJobs: 1, tools: {...} }
availability Json? // §7.10: { mode, windows, allowUrgentAnytime } — null = ALWAYS
renderNowUntil DateTime? // manual "Render Now" override; claims allowed until this time
apiKeyHash String? // optional per-machine key (dormant; §15 — shared key for now)
renderJobs RenderJob[]
heartbeats WorkerHeartbeat[]
createdAt DateTime @default(now())
updatedAt DateTime @updatedAt
@@map("machines")
}
model WorkerHeartbeat {
id String @id @default(cuid())
machineId String
machine Machine @relation(fields: [machineId], references: [id])
createdAt DateTime @default(now())
cpuPercent Float?
memPercent Float?
diskFreeGb Float?
currentJobId String?
@@index([machineId, createdAt])
@@map("worker_heartbeats")
}
model SketchTemplate { model SketchTemplate {
id String @id @default(cuid()) id String @id @default(cuid())
name String name String