feat(pipeline): standalone Blender renders on the same queue
Deploy / deploy (push) Successful in 7m2s

Blender work is not a shot submission: no Export, no version, no
validation, QC or delivery. It is "Ctrl+F12, but on the farm", so it is
modelled as what it is — a queue entry and nothing more.

That turned out to need very little. Claim, progress, fail and
complete-render already tolerated a job with no Export behind them, so a
Blender job reuses the whole lifecycle; RenderJob gains projectId, label
and submittedBy so a job with no shot code still has something to show.

Capability gating is new and was already missing: the claim now filters
job types by the machine's reported tools. Without it a workstation
without Blender would claim Blender work and fail it repeatedly, burning
the attempts while a machine that could have rendered it sat idle. The
worker asks for Blender jobs unconditionally and the server refuses them
— trusting the worker to self-select would have left the same hole.

The runner treats a clean exit with no saved frames as a failure:
Blender exits 0 when a frame range is empty or an output path is
unwritable, and a job that silently rendered nothing is the worst
outcome available.

Standalone jobs auto-requeue on a transient failure like any other work;
delivery builds are deliberately excluded, since a half-written package
wants a human first.

Adds the Blender add-on (BlenderPanel/) and its docs. The add-on refuses
to submit an unsaved file — the farm opens the .blend from disk, so
unsaved changes would render the previous state without warning. Its
payload is pinned by a contract test, since add-on and server only ever
meet over JSON.

Worker 1.5.0. Adds migration 20260815000000_blender_render_jobs.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
twotalesanimation
2026-08-15 18:53:19 +02:00
parent bada122f42
commit 70cf45ead4
24 changed files with 1347 additions and 9 deletions
+88
View File
@@ -0,0 +1,88 @@
# VFXReview — Blender Submit
A Blender add-on that queues the open `.blend` on the studio render farm instead
of rendering it locally. Ctrl+F12, but on the farm.
---
## What a Blender job is (and is not)
Deliberately the simplest thing in the pipeline: **a queue entry and nothing
more**. It has no Export, takes no shot version, and is never validated, QC'd or
delivered. The `.blend` is expected to be ready to render — output path, format
and frame range already set — and the farm just runs it.
That is the whole difference from an After Effects export, which is a shot
submission and carries the full lifecycle behind it.
---
## Install
1. **Edit → Preferences → Add-ons → Install…** and pick `vfxreview_submit.py`.
2. Enable **Render: VFXReview — Submit Render**.
3. Expand its preferences and set:
| Setting | Value |
|---|---|
| **Server URL** | `https://review.twotalesvfx.com` |
| **API Key** | matches `API_SECRET_KEY` on the server (same key the AE panel uses) |
| **Artist Email** | recorded against the job so the queue shows who submitted it |
The panel appears under **Properties → Output → VFXReview**.
---
## Submitting
1. **Refresh Projects** (the circular arrow) and pick the project.
2. Optionally set a **Label** — defaults to the `.blend` filename, and is what
the render queue shows, since there is no shot code.
3. **Urgent** makes the job claimable outside the render window, ahead of normal
work.
4. **Submit to Render Farm**.
The panel shows the frame range, engine and output path it will send, read from
the file as saved. Nothing is changed on submission.
**The file must be saved first.** The farm opens the `.blend` from disk, so
unsaved changes would render the previous state without warning — the add-on
refuses to submit a dirty file rather than let that happen.
Track the job on the web app under **Pipeline → Blender renders**: live
progress, current frame, ETA and the machine running it, with cancel and requeue.
---
## Which machines can render it
Only workers with `blenderPath` set in their `config.json` and the executable
present. The worker reports Blender at registration and the server hands
`BLENDER_RENDER` jobs only to machines that have it — a machine without Blender
can never claim work it could only fail.
Verify on a worker with:
```powershell
.\VFXReviewWorker.exe --check
```
Blender is listed as optional there: a machine without it stays perfectly valid
for After Effects work.
---
## Notes
**Paths must resolve on the render machine.** The `.blend` path and its output
path are submitted as the artist sees them; the worker applies its own
`pathMappings` before opening the file. A path on a drive letter the render
account cannot see is the most common reason a job fails immediately.
**Renders as saved.** The add-on sends the scene's own settings. The manifest
supports overrides (samples, resolution %, device, engine) for future use, but
nothing in the UI sets them today — one place for render settings is the point.
**Exit code 0 with no frames is treated as a failure.** Blender exits cleanly
when a frame range is empty or an output path is unwritable, so the worker
checks that frames were actually saved before reporting success.
+251
View File
@@ -0,0 +1,251 @@
bl_info = {
"name": "VFXReview — Submit Render",
"author": "Two Tales",
"version": (1, 0, 0),
"blender": (3, 6, 0),
"location": "Properties > Output > VFXReview",
"description": "Queue this .blend on the studio render farm instead of rendering locally",
"category": "Render",
}
import json
import os
import urllib.error
import urllib.request
import bpy
from bpy.props import BoolProperty, EnumProperty, StringProperty
from bpy.types import AddonPreferences, Operator, Panel
# ── Preferences (per artist machine, like the AE panel's Settings) ────────────
class VFXRPreferences(AddonPreferences):
bl_idname = __name__
server_url: StringProperty(
name="Server URL",
default="https://review.twotalesvfx.com",
)
api_key: StringProperty(
name="API Key",
subtype="PASSWORD",
description="Matches API_SECRET_KEY on the server",
)
artist_email: StringProperty(
name="Artist Email",
description="Recorded against the job so the queue shows who submitted it",
)
def draw(self, context):
col = self.layout.column()
col.prop(self, "server_url")
col.prop(self, "api_key")
col.prop(self, "artist_email")
def prefs():
return bpy.context.preferences.addons[__name__].preferences
# ── API ───────────────────────────────────────────────────────────────────────
def api_request(path, payload=None, method="GET"):
"""Minimal JSON client. urllib keeps the add-on dependency-free."""
p = prefs()
if not p.server_url or not p.api_key:
raise RuntimeError("Set the server URL and API key in Add-on Preferences")
url = p.server_url.rstrip("/") + path
data = json.dumps(payload).encode("utf-8") if payload is not None else None
req = urllib.request.Request(url, data=data, method=method)
req.add_header("Authorization", "Bearer " + p.api_key)
req.add_header("Accept", "application/json")
if data:
req.add_header("Content-Type", "application/json")
try:
with urllib.request.urlopen(req, timeout=30) as res:
return json.loads(res.read().decode("utf-8"))
except urllib.error.HTTPError as e:
body = e.read().decode("utf-8", "replace")
try:
message = json.loads(body).get("error", body)
except Exception:
message = body
raise RuntimeError("Server said %s: %s" % (e.code, message))
except urllib.error.URLError as e:
raise RuntimeError("Could not reach %s (%s)" % (url, e.reason))
_project_cache = []
def project_items(self, context):
"""Project dropdown. Cached because Blender calls this on every redraw."""
if not _project_cache:
return [("", "Refresh projects…", "")]
return [(p["code"], "%s — %s" % (p["code"], p["name"]), "") for p in _project_cache]
# ── Scene → job ───────────────────────────────────────────────────────────────
def scene_summary(scene):
"""
Everything the farm needs, read from the file as saved. The .blend is
expected to be ready to render, so nothing here changes settings — it only
reports them.
"""
return {
"blendFile": bpy.data.filepath.replace("\\", "/"),
"scene": scene.name,
"frameStart": scene.frame_start,
"frameEnd": scene.frame_end,
"outputPath": bpy.path.abspath(scene.render.filepath).replace("\\", "/"),
"engine": scene.render.engine,
"blenderVersion": bpy.app.version_string,
}
class VFXR_OT_refresh_projects(Operator):
bl_idname = "vfxreview.refresh_projects"
bl_label = "Refresh Projects"
bl_description = "Load the project list from the server"
def execute(self, context):
global _project_cache
try:
data = api_request("/api/ext/projects")
except RuntimeError as err:
self.report({"ERROR"}, str(err))
return {"CANCELLED"}
_project_cache = data.get("projects", [])
self.report({"INFO"}, "%d project(s) loaded" % len(_project_cache))
return {"FINISHED"}
class VFXR_OT_submit(Operator):
bl_idname = "vfxreview.submit"
bl_label = "Submit to Render Farm"
bl_description = "Queue this .blend on the farm instead of rendering locally"
def execute(self, context):
scene = context.scene
settings = scene.vfxreview
if not bpy.data.filepath:
self.report({"ERROR"}, "Save the .blend first — the farm opens it from disk")
return {"CANCELLED"}
if bpy.data.is_dirty:
# Unsaved changes would render the previous state without warning.
self.report({"ERROR"}, "Unsaved changes — save the file before submitting")
return {"CANCELLED"}
if not settings.project_code:
self.report({"ERROR"}, "Pick a project (Refresh Projects first)")
return {"CANCELLED"}
manifest = scene_summary(scene)
if manifest["frameEnd"] < manifest["frameStart"]:
self.report({"ERROR"}, "Frame end is before frame start")
return {"CANCELLED"}
label = settings.label.strip() or os.path.splitext(os.path.basename(bpy.data.filepath))[0]
try:
result = api_request(
"/api/ext/render/blender",
{
"projectCode": settings.project_code,
"label": label,
"manifest": manifest,
"submittedByEmail": prefs().artist_email or None,
"urgent": settings.urgent,
},
method="POST",
)
except RuntimeError as err:
self.report({"ERROR"}, str(err))
return {"CANCELLED"}
job = result.get("job", {})
self.report(
{"INFO"},
"Queued '%s' — %d frames%s" % (label, job.get("frames", 0), " (urgent)" if settings.urgent else ""),
)
return {"FINISHED"}
class VFXRSceneSettings(bpy.types.PropertyGroup):
project_code: EnumProperty(name="Project", items=project_items)
label: StringProperty(
name="Label",
description="Shown in the render queue. Defaults to the .blend filename",
)
urgent: BoolProperty(
name="Urgent",
description="Claimable outside the render window, ahead of normal jobs",
default=False,
)
# ── UI ────────────────────────────────────────────────────────────────────────
class VFXR_PT_submit(Panel):
bl_label = "VFXReview"
bl_space_type = "PROPERTIES"
bl_region_type = "WINDOW"
bl_context = "output"
def draw(self, context):
scene = context.scene
settings = scene.vfxreview
col = self.layout.column()
row = col.row(align=True)
row.prop(settings, "project_code", text="")
row.operator("vfxreview.refresh_projects", text="", icon="FILE_REFRESH")
col.prop(settings, "label")
col.prop(settings, "urgent")
box = col.box()
box.label(text="Renders as saved:", icon="INFO")
box.label(text="Frames %d–%d (%d)" % (
scene.frame_start, scene.frame_end, scene.frame_end - scene.frame_start + 1))
box.label(text="Engine: %s" % scene.render.engine)
output = bpy.path.abspath(scene.render.filepath)
box.label(text="Output: %s" % (output[-48:] if len(output) > 48 else output))
if bpy.data.is_dirty or not bpy.data.filepath:
warn = col.row()
warn.alert = True
warn.label(text="Save the file before submitting", icon="ERROR")
col.separator()
col.operator("vfxreview.submit", icon="RENDER_ANIMATION")
classes = (
VFXRPreferences,
VFXRSceneSettings,
VFXR_OT_refresh_projects,
VFXR_OT_submit,
VFXR_PT_submit,
)
def register():
for cls in classes:
bpy.utils.register_class(cls)
bpy.types.Scene.vfxreview = bpy.props.PointerProperty(type=VFXRSceneSettings)
def unregister():
del bpy.types.Scene.vfxreview
for cls in reversed(classes):
bpy.utils.unregister_class(cls)
if __name__ == "__main__":
register()
+1
View File
@@ -174,6 +174,7 @@ prisma/ — schema.prisma, seed.ts
| [DELIVERY_WORKFLOW.md](DELIVERY_WORKFLOW.md) | Building and shipping delivery packages (how-to) | | [DELIVERY_WORKFLOW.md](DELIVERY_WORKFLOW.md) | Building and shipping delivery packages (how-to) |
| [VFXReviewConnector.md](VFXReviewConnector.md) | After Effects panel — setup, Shot Builder, QC, diagnostics | | [VFXReviewConnector.md](VFXReviewConnector.md) | After Effects panel — setup, Shot Builder, QC, diagnostics |
| [RenderWorker/README.md](RenderWorker/README.md) | Render worker service — packaging, install, validation, delivery builds | | [RenderWorker/README.md](RenderWorker/README.md) | Render worker service — packaging, install, validation, delivery builds |
| [BlenderPanel/README.md](BlenderPanel/README.md) | Blender add-on — queueing a .blend on the farm |
| [EXT_API_REFERENCE.md](EXT_API_REFERENCE.md) | External API (`/api/ext/*`) used by the panel and workers | | [EXT_API_REFERENCE.md](EXT_API_REFERENCE.md) | External API (`/api/ext/*`) used by the panel and workers |
| [Shot task workflow.md](Shot%20task%20workflow.md) | Shot and task lifecycle in the web app | | [Shot task workflow.md](Shot%20task%20workflow.md) | Shot and task lifecycle in the web app |
+18
View File
@@ -89,6 +89,23 @@ Server-side switches (SystemConfig, no worker redeploy): `validation.enabled`,
`validation.sampleEvery`, `validation.handleFrames`, and `validation.overrides` for per-check `validation.sampleEvery`, `validation.handleFrames`, and `validation.overrides` for per-check
severity, e.g. `{"exr_type":"warn"}`. severity, e.g. `{"exr_type":"warn"}`.
## Blender renders
The worker also claims `BLENDER_RENDER` jobs: standalone renders submitted from
the Blender add-on, with no Export behind them — nothing to validate, preview or
QC. It runs `blender -b file.blend -s N -e M -a`, parses `Fra:`/`Saved:` output
for progress, and reports complete or failed like any other job.
Set `blenderPath` in `config.json` to make a machine eligible. The version is
reported at registration and **the server only hands Blender jobs to machines
that have it**, so a workstation without Blender can never claim work it could
only fail. A machine without it stays perfectly valid for After Effects work.
Blender exits 0 when a frame range is empty or an output path is unwritable, so
a run that saves no frames is reported as a failure rather than a success.
See [BlenderPanel/README.md](../BlenderPanel/README.md).
## Delivery builds ## Delivery builds
The same service also claims `DELIVERY_BUILD` jobs: creating a dated package The same service also claims `DELIVERY_BUILD` jobs: creating a dated package
@@ -174,6 +191,7 @@ Create `C:\ProgramData\VFXReviewWorker\config.json` (§7.8):
"aerenderPath": "C:\\Program Files\\Adobe\\Adobe After Effects 2026\\Support Files\\aerender.exe", "aerenderPath": "C:\\Program Files\\Adobe\\Adobe After Effects 2026\\Support Files\\aerender.exe",
"aeVersion": "24.3", "aeVersion": "24.3",
"ffmpegPath": "C:\\pipeline\\bin\\ffmpeg.exe", "ffmpegPath": "C:\\pipeline\\bin\\ffmpeg.exe",
"blenderPath": "C:\\Program Files\\Blender Foundation\\Blender 4.2\\blender.exe",
"plateSearchRoots": ["V:/_FOOTAGE/UNG"], "plateSearchRoots": ["V:/_FOOTAGE/UNG"],
"pathMappings": [ "pathMappings": [
{ "from": "//SAN/", "to": "S:/" } { "from": "//SAN/", "to": "S:/" }
+14 -1
View File
@@ -37,7 +37,20 @@ public sealed class ApiClient
hostname = Dns.GetHostName(), hostname = Dns.GetHostName(),
workerVersion = o.WorkerVersion, workerVersion = o.WorkerVersion,
aeVersion = o.AeVersion, aeVersion = o.AeVersion,
capabilities = new { maxConcurrentJobs = 1, tools = new { ffmpeg = o.FfmpegPath != null, oiiotool = o.OiiotoolPath != null } }, // The server hands out BLENDER_RENDER work on the strength of the
// blender entry, so it reports the resolved path rather than a bare
// flag: a machine claiming Blender jobs it cannot run is worse than
// one that never claims them.
capabilities = new
{
maxConcurrentJobs = 1,
tools = new
{
ffmpeg = o.FfmpegPath != null,
oiiotool = o.OiiotoolPath != null,
blender = !string.IsNullOrWhiteSpace(o.BlenderPath) && File.Exists(o.BlenderPath) ? o.BlenderPath : null,
},
},
}, Json, ct); }, Json, ct);
res.EnsureSuccessStatusCode(); res.EnsureSuccessStatusCode();
return (await res.Content.ReadFromJsonAsync<RegisterResponse>(Json, ct))!; return (await res.Content.ReadFromJsonAsync<RegisterResponse>(Json, ct))!;
@@ -0,0 +1,68 @@
using System.Text.RegularExpressions;
namespace VFXReviewWorker;
/// <summary>
/// Parses `blender -b` stdout. Blender reports the frame it is working on in
/// its status lines ("Fra:1042 Mem:512.00M ...") and confirms each written file
/// with "Saved: '...'". The saved lines are the reliable signal — a frame can
/// be reported for a long time while it renders — so progress counts those and
/// falls back to Fra: for a first sign of life.
/// </summary>
public static partial class BlenderProgressParser
{
[GeneratedRegex(@"^Fra:(\d+)", RegexOptions.Compiled)]
private static partial Regex FrameLine();
[GeneratedRegex(@"^Saved:\s*'(.+)'", RegexOptions.Compiled)]
private static partial Regex SavedLine();
[GeneratedRegex(@"^(Error|Warning): |^Error: ", RegexOptions.Compiled | RegexOptions.IgnoreCase)]
private static partial Regex ErrorLine();
/// <summary>
/// Failures that will repeat identically on another machine — a missing
/// file, an unknown scene, an unwritable output. Retrying these burns the
/// queue for nothing; a human has to fix the .blend or the path.
/// </summary>
private static readonly string[] NonRetryablePatterns =
{
"cannot read file",
"file not found",
"unable to open",
"is not a blend file",
"failed to read blend file",
"unknown scene",
"scene not found",
"unable to create directory",
"cannot write file",
"no camera",
"python script fail",
"unknown argument",
};
/// <summary>The frame Blender is currently working on, or null.</summary>
public static int? ParseFrame(string line)
{
var m = FrameLine().Match(line);
return m.Success && int.TryParse(m.Groups[1].Value, out var f) ? f : null;
}
/// <summary>The path of a frame Blender has just written, or null.</summary>
public static string? ParseSaved(string line)
{
var m = SavedLine().Match(line);
return m.Success ? m.Groups[1].Value : null;
}
public static bool IsErrorLine(string line) => ErrorLine().IsMatch(line);
public static bool IsNonRetryableError(string line)
{
foreach (var p in NonRetryablePatterns)
{
if (line.Contains(p, StringComparison.OrdinalIgnoreCase)) return true;
}
return false;
}
}
@@ -0,0 +1,260 @@
using System.Diagnostics;
using System.Text;
using Microsoft.Extensions.Logging;
namespace VFXReviewWorker;
/// <summary>
/// Standalone Blender renders: `blender -b file.blend -s N -e M -a`.
///
/// The .blend is expected to be ready to render — output path, format and
/// frame range already set by whoever built it. Overrides are only applied
/// when the submission asked for them, so this stays "Ctrl+F12 on the farm"
/// rather than a second place where render settings can drift.
/// </summary>
public sealed class BlenderRunner
{
private readonly WorkerOptions _options;
private readonly ApiClient _api;
private readonly ILogger<BlenderRunner> _log;
public BlenderRunner(WorkerOptions options, ApiClient api, ILogger<BlenderRunner> log)
{
_options = options;
_api = api;
_log = log;
}
public async Task<RenderResult> RunAsync(
ClaimedJob job,
BlenderManifest m,
string machineId,
ServerConfig config,
CancellationToken cancelJob,
CancellationToken shutdown)
{
var mapper = new PathMapper(_options.PathMappings);
var blendLocal = mapper.Map(m.BlendFile);
var totalFrames = m.TotalFrames;
if (!File.Exists(blendLocal))
{
return new RenderResult(false, -1, Retryable: false,
$"Blend file not found on this machine: {blendLocal}", "", "", 0);
}
// Argument order matters to Blender: everything before -a configures the
// render, and -a must come last or it renders with the old settings.
var args = new List<string> { "-b", blendLocal };
if (!string.IsNullOrWhiteSpace(m.Scene)) { args.Add("-S"); args.Add(m.Scene!); }
if (!string.IsNullOrWhiteSpace(m.OutputPath))
{
args.Add("-o");
args.Add(mapper.Map(m.OutputPath!));
}
if (!string.IsNullOrWhiteSpace(m.Engine)) { args.Add("-E"); args.Add(m.Engine!); }
var pythonOverrides = BuildOverrideExpression(m);
if (pythonOverrides is not null) { args.Add("--python-expr"); args.Add(pythonOverrides); }
args.Add("-s"); args.Add(m.FrameStart.ToString());
args.Add("-e"); args.Add(m.FrameEnd.ToString());
args.Add("-a");
_log.LogInformation("Rendering {Label}: {Blend} frames {Start}-{End}",
job.Label ?? job.Id, blendLocal, m.FrameStart, m.FrameEnd);
return await ExecuteAsync(job.Id, machineId, args, totalFrames, m.FrameStart, config,
$"blender_{job.Id}.log", cancelJob, shutdown);
}
/// <summary>
/// Settings Blender has no command-line flag for. Returns null when the
/// submission asked for no overrides, so the file renders exactly as saved.
/// </summary>
private static string? BuildOverrideExpression(BlenderManifest m)
{
var parts = new List<string>();
if (m.Samples is int samples)
{
parts.Add($"getattr(S.cycles, 'samples', None) and setattr(S.cycles, 'samples', {samples})");
}
if (m.ResolutionPercent is int pct)
{
parts.Add($"setattr(S.render, 'resolution_percentage', {pct})");
}
if (!string.IsNullOrWhiteSpace(m.Device))
{
var deviceValue = m.Device!.Equals("GPU", StringComparison.OrdinalIgnoreCase) ? "GPU" : "CPU";
parts.Add($"hasattr(S, 'cycles') and setattr(S.cycles, 'device', '{deviceValue}')");
}
if (parts.Count == 0) return null;
return "import bpy; S = bpy.context.scene; " + string.Join("; ", parts);
}
private async Task<RenderResult> ExecuteAsync(
string jobId,
string machineId,
List<string> args,
int totalFrames,
int frameStart,
ServerConfig config,
string logFileName,
CancellationToken cancelJob,
CancellationToken shutdown)
{
var fullLogPath = Path.Combine(WorkerOptions.LogsDir, logFileName);
Directory.CreateDirectory(WorkerOptions.LogsDir);
var psi = new ProcessStartInfo
{
FileName = _options.BlenderPath,
RedirectStandardOutput = true,
RedirectStandardError = true,
UseShellExecute = false,
CreateNoWindow = true,
StandardOutputEncoding = Encoding.UTF8,
StandardErrorEncoding = Encoding.UTF8,
};
foreach (var a in args) psi.ArgumentList.Add(a);
var tail = new LogTail(200);
var eta = new EtaCalculator();
var sw = Stopwatch.StartNew();
var lastProgressAt = DateTimeOffset.UtcNow;
var lastReportAt = DateTimeOffset.MinValue;
var framesSaved = 0;
var currentFrame = frameStart;
string? firstError = null;
var nonRetryable = false;
var cancelledByServer = false;
using var process = new Process { StartInfo = psi };
await using var fullLog = new StreamWriter(fullLogPath, append: false, Encoding.UTF8);
var logLock = new object();
void HandleLine(string? line, bool isStderr)
{
if (line is null) return;
lock (logLock)
{
fullLog.WriteLine(line);
tail.Add(line);
}
var frame = BlenderProgressParser.ParseFrame(line);
if (frame is not null)
{
// A frame line only proves Blender is alive; a long frame emits
// many of them, so this feeds the stall watchdog, not progress.
currentFrame = frame.Value;
lastProgressAt = DateTimeOffset.UtcNow;
}
if (BlenderProgressParser.ParseSaved(line) is not null)
{
framesSaved++;
lastProgressAt = DateTimeOffset.UtcNow;
eta.RecordFrame(lastProgressAt);
}
if ((isStderr || BlenderProgressParser.IsErrorLine(line)) && !string.IsNullOrWhiteSpace(line))
{
if (BlenderProgressParser.IsErrorLine(line)) firstError ??= line.Trim();
if (BlenderProgressParser.IsNonRetryableError(line)) nonRetryable = true;
}
}
process.OutputDataReceived += (_, e) => HandleLine(e.Data, isStderr: false);
process.ErrorDataReceived += (_, e) => HandleLine(e.Data, isStderr: true);
try
{
process.Start();
}
catch (Exception ex)
{
return new RenderResult(false, -1, Retryable: false,
$"Failed to launch Blender at {_options.BlenderPath}: {ex.Message}",
tail.ToString(), fullLogPath, 0);
}
process.BeginOutputReadLine();
process.BeginErrorReadLine();
while (!process.HasExited)
{
await Task.Delay(TimeSpan.FromSeconds(2), CancellationToken.None);
if (cancelJob.IsCancellationRequested || shutdown.IsCancellationRequested)
{
_log.LogWarning("Cancellation requested — killing Blender tree for job {JobId}", jobId);
KillTree(process);
cancelledByServer = cancelJob.IsCancellationRequested;
break;
}
if ((DateTimeOffset.UtcNow - lastProgressAt).TotalSeconds > config.StallTimeoutSeconds)
{
_log.LogError("Blender stalled ({Sec}s without output) — killing job {JobId}", config.StallTimeoutSeconds, jobId);
KillTree(process);
await WaitForExitSafe(process);
return new RenderResult(false, -2, Retryable: true,
$"Blender produced no output for {config.StallTimeoutSeconds}s (stall timeout)",
tail.ToString(), fullLogPath, sw.Elapsed.TotalSeconds);
}
if ((DateTimeOffset.UtcNow - lastReportAt).TotalSeconds >= 10)
{
lastReportAt = DateTimeOffset.UtcNow;
var progress = totalFrames > 0 ? Math.Min(1.0, (double)framesSaved / totalFrames) : 0;
var resp = await _api.ProgressAsync(jobId, machineId, progress, currentFrame, totalFrames,
eta.EstimateSeconds(totalFrames - framesSaved), tail.ToString(), shutdown);
if (resp?.CancelRequested == true)
{
_log.LogWarning("Server requested cancel for job {JobId}", jobId);
KillTree(process);
cancelledByServer = true;
break;
}
}
}
await WaitForExitSafe(process);
sw.Stop();
lock (logLock) { fullLog.Flush(); }
if (cancelledByServer)
{
return new RenderResult(false, -3, Retryable: false, "Cancelled", tail.ToString(), fullLogPath, sw.Elapsed.TotalSeconds);
}
var exitCode = process.ExitCode;
if (exitCode != 0)
{
return new RenderResult(false, exitCode, Retryable: !nonRetryable,
firstError ?? $"Blender exited with code {exitCode}",
tail.ToString(), fullLogPath, sw.Elapsed.TotalSeconds);
}
// Blender exits 0 having rendered nothing if the range is empty or the
// output path is unwritable, so a silent success is treated as failure.
if (framesSaved == 0)
{
return new RenderResult(false, 0, Retryable: false,
"Blender exited cleanly but saved no frames — check the output path and frame range in the .blend",
tail.ToString(), fullLogPath, sw.Elapsed.TotalSeconds);
}
_log.LogInformation("Job {JobId}: {Saved}/{Total} frames in {Sec:F0}s", jobId, framesSaved, totalFrames, sw.Elapsed.TotalSeconds);
return new RenderResult(true, 0, true, null, tail.ToString(), fullLogPath, sw.Elapsed.TotalSeconds);
}
private static void KillTree(Process process)
{
try { if (!process.HasExited) process.Kill(entireProcessTree: true); } catch { }
}
private static async Task WaitForExitSafe(Process process)
{
try { await process.WaitForExitAsync(); } catch { }
}
}
+29
View File
@@ -49,6 +49,9 @@ public sealed class ClaimedJob
public int Attempt { get; set; } public int Attempt { get; set; }
public int MaxAttempts { get; set; } public int MaxAttempts { get; set; }
public string? ExportId { get; set; } public string? ExportId { get; set; }
public string? DeliveryId { get; set; }
/// <summary>Set for jobs with no shot behind them; shown instead of a shot code.</summary>
public string? Label { get; set; }
public DateTimeOffset? LeaseExpiresAt { get; set; } public DateTimeOffset? LeaseExpiresAt { get; set; }
/// <summary>Manifest shape depends on the job type — deserialize per stage.</summary> /// <summary>Manifest shape depends on the job type — deserialize per stage.</summary>
@@ -65,6 +68,32 @@ public sealed class ClaimedJob
public DeliveryManifest AsDeliveryManifest() => public DeliveryManifest AsDeliveryManifest() =>
Manifest.Deserialize<DeliveryManifest>(WorkerOptions.JsonOpts) Manifest.Deserialize<DeliveryManifest>(WorkerOptions.JsonOpts)
?? throw new InvalidOperationException("Job manifest is not a delivery manifest"); ?? throw new InvalidOperationException("Job manifest is not a delivery manifest");
public BlenderManifest AsBlenderManifest() =>
Manifest.Deserialize<BlenderManifest>(WorkerOptions.JsonOpts)
?? throw new InvalidOperationException("Job manifest is not a Blender manifest");
}
/// <summary>
/// A standalone Blender render. Overrides are all optional: an unset value
/// leaves whatever the .blend was saved with, which is the point — the file is
/// expected to be ready to render.
/// </summary>
public sealed class BlenderManifest
{
public string Stage { get; set; } = "BLENDER";
public string BlendFile { get; set; } = "";
public string? Scene { get; set; }
public int FrameStart { get; set; }
public int FrameEnd { get; set; }
public string? OutputPath { get; set; }
public string? Engine { get; set; }
public string? Device { get; set; }
public int? Samples { get; set; }
public int? ResolutionPercent { get; set; }
public string? BlenderVersion { get; set; }
public int TotalFrames => FrameEnd - FrameStart + 1;
} }
/// <summary>DELIVERY_BUILD manifest (server-built, RenderPipeline2 §11).</summary> /// <summary>DELIVERY_BUILD manifest (server-built, RenderPipeline2 §11).</summary>
+77 -1
View File
@@ -16,11 +16,12 @@ public sealed class JobExecutor
private readonly PreviewStage _preview; private readonly PreviewStage _preview;
private readonly ValidationStage _validation; private readonly ValidationStage _validation;
private readonly DeliveryStage _delivery; private readonly DeliveryStage _delivery;
private readonly BlenderRunner _blender;
private readonly ReportSpooler _spooler; private readonly ReportSpooler _spooler;
private readonly ILogger<JobExecutor> _log; private readonly ILogger<JobExecutor> _log;
public JobExecutor(WorkerOptions options, ApiClient api, AerenderRunner runner, PreviewStage preview, public JobExecutor(WorkerOptions options, ApiClient api, AerenderRunner runner, PreviewStage preview,
ValidationStage validation, DeliveryStage delivery, ReportSpooler spooler, ILogger<JobExecutor> log) ValidationStage validation, DeliveryStage delivery, BlenderRunner blender, ReportSpooler spooler, ILogger<JobExecutor> log)
{ {
_options = options; _options = options;
_api = api; _api = api;
@@ -28,6 +29,7 @@ public sealed class JobExecutor
_preview = preview; _preview = preview;
_validation = validation; _validation = validation;
_delivery = delivery; _delivery = delivery;
_blender = blender;
_spooler = spooler; _spooler = spooler;
_log = log; _log = log;
} }
@@ -46,6 +48,12 @@ public sealed class JobExecutor
return; return;
} }
if (job.Type == "BLENDER_RENDER")
{
await RunBlenderAsync(job, machineId, config, cancelJob, shutdown);
return;
}
var manifest = job.AsRenderManifest(); var manifest = job.AsRenderManifest();
var mapper = new PathMapper(_options.PathMappings); var mapper = new PathMapper(_options.PathMappings);
var outputDirLocal = mapper.Map(manifest.OutputDir); var outputDirLocal = mapper.Map(manifest.OutputDir);
@@ -169,6 +177,74 @@ public sealed class JobExecutor
} }
} }
/// <summary>
/// BLENDER_RENDER job: a standalone render with no Export behind it, so
/// there is nothing to validate, preview or QC — it finishes at
/// complete-render and that is the whole lifecycle.
/// </summary>
private async Task RunBlenderAsync(ClaimedJob job, string machineId, ServerConfig config,
CancellationToken cancelJob, CancellationToken shutdown)
{
var manifest = job.AsBlenderManifest();
WriteState(new CurrentJobState
{
JobId = job.Id,
ExportId = null,
MachineId = machineId,
OutputDirLocal = manifest.OutputPath ?? "",
ClaimedAt = DateTimeOffset.UtcNow,
});
try
{
var result = await _blender.RunAsync(job, manifest, machineId, config, cancelJob, shutdown);
var logKey = string.IsNullOrEmpty(result.FullLogPath)
? null
: await _api.UploadLogAsync(job.Id, machineId, result.FullLogPath, shutdown);
if (result.Success)
{
_log.LogInformation("Blender job {JobId} complete: {Frames} frames in {Sec:F0}s",
job.Id, manifest.TotalFrames, result.RenderSeconds);
_spooler.Enqueue("complete-render", job.Id, new Dictionary<string, object?>
{
["machineId"] = machineId,
["renderSeconds"] = (int)result.RenderSeconds,
["logFileKey"] = logKey,
["logTail"] = result.LogTail,
});
return;
}
_spooler.Enqueue("fail", job.Id, new Dictionary<string, object?>
{
["machineId"] = machineId,
["stage"] = "BLENDER",
["exitCode"] = result.ExitCode,
["errorMessage"] = result.ErrorMessage,
["logTail"] = result.LogTail,
["logFileKey"] = logKey,
["retryable"] = result.ErrorMessage == "Cancelled" ? false : result.Retryable,
});
}
catch (Exception ex)
{
_log.LogError(ex, "Unexpected Blender error for job {JobId}", job.Id);
_spooler.Enqueue("fail", job.Id, new Dictionary<string, object?>
{
["machineId"] = machineId,
["stage"] = "BLENDER",
["errorMessage"] = $"Worker exception: {ex.Message}",
["retryable"] = true,
});
}
finally
{
ClearState();
}
}
/// <summary> /// <summary>
/// DELIVERY_BUILD job: build the dated package folder from already-QC'd /// DELIVERY_BUILD job: build the dated package folder from already-QC'd
/// exports. No AE involved — folders, links and verification only. /// exports. No AE involved — folders, links and verification only.
@@ -38,6 +38,11 @@ public static class Preflight
Console.WriteLine(); Console.WriteLine();
ok &= Report("aerender.exe", options.AerenderPath, File.Exists(options.AerenderPath)); ok &= Report("aerender.exe", options.AerenderPath, File.Exists(options.AerenderPath));
// Not required: a machine without Blender simply never claims Blender
// jobs, so this reports rather than fails.
Report("blender.exe", options.BlenderPath ?? "(not configured)",
!string.IsNullOrWhiteSpace(options.BlenderPath) && File.Exists(options.BlenderPath),
"needed only for Blender render jobs");
var afterFx = options.ResolveAfterFxPath(); var afterFx = options.ResolveAfterFxPath();
ok &= Report("AfterFX.com", afterFx, File.Exists(afterFx), ok &= Report("AfterFX.com", afterFx, File.Exists(afterFx),
"needed for preview builds only"); "needed for preview builds only");
+1
View File
@@ -41,6 +41,7 @@ builder.Services.AddSingleton<AerenderRunner>();
builder.Services.AddSingleton<PreviewStage>(); builder.Services.AddSingleton<PreviewStage>();
builder.Services.AddSingleton<ValidationStage>(); builder.Services.AddSingleton<ValidationStage>();
builder.Services.AddSingleton<DeliveryStage>(); builder.Services.AddSingleton<DeliveryStage>();
builder.Services.AddSingleton<BlenderRunner>();
builder.Services.AddSingleton<JobExecutor>(); builder.Services.AddSingleton<JobExecutor>();
builder.Services.AddHostedService<WorkerService>(); builder.Services.AddHostedService<WorkerService>();
@@ -15,7 +15,7 @@
every change that gets published. publish.ps1 appends the git SHA via every change that gets published. publish.ps1 appends the git SHA via
SourceRevisionId, giving "1.1.0+feb1c7b". SourceRevisionId, giving "1.1.0+feb1c7b".
--> -->
<Version>1.4.0</Version> <Version>1.5.0</Version>
</PropertyGroup> </PropertyGroup>
<ItemGroup> <ItemGroup>
@@ -20,6 +20,12 @@ public sealed class WorkerOptions
/// <summary>vfxr_build_preview.jsx — defaults to scripts\ beside the worker exe.</summary> /// <summary>vfxr_build_preview.jsx — defaults to scripts\ beside the worker exe.</summary>
public string? PreviewScriptPath { get; set; } public string? PreviewScriptPath { get; set; }
public string? FfmpegPath { get; set; } public string? FfmpegPath { get; set; }
/// <summary>
/// blender.exe. Its presence is what makes this machine eligible for
/// Blender jobs: the version is reported at registration and the server
/// only hands BLENDER_RENDER work to machines that have it.
/// </summary>
public string? BlenderPath { get; set; }
public string? OiiotoolPath { get; set; } public string? OiiotoolPath { get; set; }
public List<PathMapping> PathMappings { get; set; } = new(); public List<PathMapping> PathMappings { get; set; } = new();
/// <summary> /// <summary>
@@ -64,9 +64,11 @@ public sealed class WorkerService : BackgroundService
} }
// Delivery builds are file work with no AE involvement, so they // Delivery builds are file work with no AE involvement, so they
// are always claimable — someone is usually waiting on the folder. // are always claimable — someone is usually waiting on the folder.
// Blender is asked for unconditionally; the server refuses it to
// machines whose capabilities do not include Blender.
var types = aeOpen var types = aeOpen
? new[] { "AE_RENDER", "DELIVERY_BUILD" } ? new[] { "AE_RENDER", "DELIVERY_BUILD", "BLENDER_RENDER" }
: new[] { "AE_RENDER", "PREVIEW_ONLY", "DELIVERY_BUILD" }; : new[] { "AE_RENDER", "PREVIEW_ONLY", "DELIVERY_BUILD", "BLENDER_RENDER" };
var job = await _api.ClaimAsync(_machineId!, types, stoppingToken); var job = await _api.ClaimAsync(_machineId!, types, stoppingToken);
if (job is not null) if (job is not null)
@@ -112,6 +112,12 @@ export function PipelineQueueClient({ projects }: Props) {
</p> </p>
</div> </div>
<div className="flex items-center gap-2"> <div className="flex items-center gap-2">
<Link href="/pipeline/deliveries">
<Button variant="outline" size="sm">
<Package className="h-4 w-4 mr-2" />
Deliveries
</Button>
</Link>
<Link href="/pipeline/deliveries"> <Link href="/pipeline/deliveries">
<Button variant="outline" size="sm"> <Button variant="outline" size="sm">
<Package className="h-4 w-4 mr-2" /> <Package className="h-4 w-4 mr-2" />
@@ -274,6 +280,124 @@ export function PipelineQueueClient({ projects }: Props) {
</tbody> </tbody>
</table> </table>
</div> </div>
<BlenderQueue projectId={projectId} />
</div>
);
}
interface BlenderJobRow {
id: string;
label: string | null;
status: string;
attempt: number;
maxAttempts: number;
progress: number;
currentFrame: number | null;
totalFrames: number | null;
etaSeconds: number | null;
errorMessage: string | null;
machineName: string | null;
submittedBy: string | null;
projectCode: string | null;
blendFile: string | null;
frameStart: number | null;
frameEnd: number | null;
createdAt: string;
}
const BLENDER_STATUS_STYLES: Record<string, string> = {
QUEUED: "bg-zinc-500/15 text-zinc-300 border-zinc-500/30",
CLAIMED: "bg-blue-500/15 text-blue-400 border-blue-500/30",
RUNNING: "bg-blue-500/15 text-blue-400 border-blue-500/30",
COMPLETED: "bg-emerald-500/15 text-emerald-400 border-emerald-500/30",
FAILED: "bg-red-500/15 text-red-400 border-red-500/30",
CANCELLED: "bg-zinc-500/10 text-zinc-500 border-zinc-600/30",
EXPIRED: "bg-red-500/15 text-red-400 border-red-500/30",
};
/**
* Standalone Blender renders. They have no Export behind them, so they cannot
* appear in the table above — but they share the queue, and a render nobody can
* see is a render nobody can retry.
*/
function BlenderQueue({ projectId }: { projectId: string }) {
const { toast } = useToast();
const queryClient = useQueryClient();
const { data } = useQuery({
queryKey: ["blender-jobs", projectId],
queryFn: async () => {
const qs = projectId && projectId !== "all" ? `?projectId=${encodeURIComponent(projectId)}` : "";
const res = await fetch(`/api/render/blender${qs}`);
if (!res.ok) throw new Error("Failed to load Blender jobs");
return res.json() as Promise<{ jobs: BlenderJobRow[] }>;
},
refetchInterval: 5000,
});
async function jobAction(id: string, verb: "cancel" | "retry") {
const res = await fetch(`/api/render/blender/${id}?action=${verb}`, { method: "POST" });
if (!res.ok) {
const body = await res.json().catch(() => ({}));
toast({ title: `${verb} failed`, description: body.error ?? res.statusText, variant: "destructive" });
}
queryClient.invalidateQueries({ queryKey: ["blender-jobs"] });
}
const jobs = data?.jobs ?? [];
if (jobs.length === 0) return null;
return (
<div className="rounded-lg border border-zinc-800 bg-zinc-900/40">
<div className="px-4 py-2.5 border-b border-zinc-800 text-sm font-medium text-white">
Blender renders
</div>
<div className="divide-y divide-zinc-800/60">
{jobs.map((j) => (
<div key={j.id} className="px-4 py-3 text-sm">
<div className="flex items-center gap-3 flex-wrap">
<span className="text-zinc-200">{j.label ?? "(untitled)"}</span>
<span className={cn("rounded-full border px-2 py-0.5 text-xs", BLENDER_STATUS_STYLES[j.status] ?? "")}>
{j.status.toLowerCase()}
</span>
{j.projectCode && <span className="text-xs text-zinc-500">{j.projectCode}</span>}
{j.frameStart != null && (
<span className="text-xs text-zinc-500">
frames {j.frameStart}–{j.frameEnd}
</span>
)}
{j.status === "RUNNING" && (
<span className="text-xs text-blue-400">
{Math.round(j.progress * 100)}%
{j.currentFrame != null ? ` · frame ${j.currentFrame}` : ""}
{j.etaSeconds ? ` · ETA ${formatEta(j.etaSeconds)}` : ""}
</span>
)}
{j.machineName && <span className="text-xs text-zinc-500">{j.machineName}</span>}
{j.attempt > 1 && (
<span className="text-xs text-zinc-500">
attempt {j.attempt}/{j.maxAttempts}
</span>
)}
<span className="ml-auto flex items-center gap-1">
{["QUEUED", "CLAIMED", "RUNNING"].includes(j.status) && (
<Button variant="ghost" size="icon-sm" title="Cancel" onClick={() => jobAction(j.id, "cancel")}>
<XCircle className="h-4 w-4 text-red-400" />
</Button>
)}
{["FAILED", "CANCELLED", "EXPIRED"].includes(j.status) && (
<Button variant="ghost" size="icon-sm" title="Requeue" onClick={() => jobAction(j.id, "retry")}>
<RotateCcw className="h-4 w-4 text-amber-400" />
</Button>
)}
</span>
</div>
{j.blendFile && <div className="mt-1 text-xs text-zinc-600 break-all">{j.blendFile}</div>}
{j.errorMessage && <div className="mt-1 text-xs text-red-400">{j.errorMessage}</div>}
</div>
))}
</div>
</div> </div>
); );
} }
+36
View File
@@ -0,0 +1,36 @@
import { NextRequest, NextResponse } from "next/server";
import { isExtAuthorized } from "@/lib/render-pipeline/ext-auth";
import { toErrorResponse } from "@/lib/render-pipeline/errors";
import { createBlenderJob, createBlenderJobSchema } from "@/lib/render-pipeline/blender";
// ── POST /api/ext/render/blender ─────────────────────────────────────────────
//
// Queue a standalone Blender render from the Blender add-on. No Export, no shot
// version, no validation or QC — the .blend is already set up to render, and
// this puts it on the farm instead of on the artist's machine.
export async function POST(req: NextRequest) {
if (!isExtAuthorized(req)) {
return NextResponse.json({ error: "Unauthorized" }, { status: 401 });
}
let raw: unknown;
try {
raw = await req.json();
} catch {
return NextResponse.json({ error: "Invalid JSON" }, { status: 400 });
}
const parsed = createBlenderJobSchema.safeParse(raw);
if (!parsed.success) {
return NextResponse.json({ error: "Invalid Blender job", details: parsed.error.issues }, { status: 422 });
}
try {
const result = await createBlenderJob(parsed.data);
return NextResponse.json(result, { status: 201 });
} catch (err) {
const { body, status } = toErrorResponse(err);
return NextResponse.json(body, { status });
}
}
+28
View File
@@ -0,0 +1,28 @@
import { NextRequest, NextResponse } from "next/server";
import { toErrorResponse } from "@/lib/render-pipeline/errors";
import { requirePipelineUser } from "@/lib/render-pipeline/session-auth";
import { cancelBlenderJob, retryBlenderJob } from "@/lib/render-pipeline/blender";
// ── POST /api/render/blender/{jobId}?action=cancel|retry ─────────────────────
export async function POST(
req: NextRequest,
{ params }: { params: Promise<{ jobId: string }> }
) {
try {
const user = await requirePipelineUser();
const { jobId } = await params;
const action = new URL(req.url).searchParams.get("action");
if (action === "retry") {
return NextResponse.json(await retryBlenderJob(jobId));
}
if (action === "cancel") {
return NextResponse.json(await cancelBlenderJob(jobId, user.id));
}
return NextResponse.json({ error: "action must be cancel or retry" }, { status: 400 });
} catch (err) {
const { body, status } = toErrorResponse(err);
return NextResponse.json(body, { status });
}
}
+24
View File
@@ -0,0 +1,24 @@
import { NextRequest, NextResponse } from "next/server";
import { RenderJobStatus } from "@prisma/client";
import { toErrorResponse } from "@/lib/render-pipeline/errors";
import { requirePipelineUser } from "@/lib/render-pipeline/session-auth";
import { listBlenderJobs } from "@/lib/render-pipeline/blender";
// ── GET /api/render/blender?projectId=&status= ───────────────────────────────
export async function GET(req: NextRequest) {
try {
await requirePipelineUser();
const { searchParams } = new URL(req.url);
const status = searchParams.get("status");
const result = await listBlenderJobs({
projectId: searchParams.get("projectId") ?? undefined,
status: status ? (status as RenderJobStatus) : undefined,
limit: searchParams.get("limit") ? Number(searchParams.get("limit")) : undefined,
});
return NextResponse.json(result);
} catch (err) {
const { body, status } = toErrorResponse(err);
return NextResponse.json(body, { status });
}
}
+50
View File
@@ -0,0 +1,50 @@
import { describe, it, expect } from "vitest";
import { blenderManifestSchema, createBlenderJobSchema } from "@/lib/render-pipeline/blender";
/**
* The add-on and the server are separate codebases that only meet over JSON,
* so the exact payload the add-on builds is pinned here. A field renamed on
* either side otherwise surfaces as a 422 in front of an artist.
*/
describe("Blender submission contract", () => {
// Verbatim shape of scene_summary() in BlenderPanel/vfxreview_submit.py
const addonManifest = {
blendFile: "X:/shows/UNG/cg/UNG_110_030_010_smoke.blend",
scene: "Scene",
frameStart: 1001,
frameEnd: 1110,
outputPath: "X:/shows/UNG/cg/renders/smoke/frame_",
engine: "CYCLES",
blenderVersion: "4.2.1",
};
it("accepts what the add-on sends", () => {
const parsed = blenderManifestSchema.safeParse(addonManifest);
expect(parsed.success).toBe(true);
});
it("accepts the full submission body", () => {
const parsed = createBlenderJobSchema.safeParse({
projectCode: "UNG_S1",
label: "UNG_110_030_010_smoke",
manifest: addonManifest,
submittedByEmail: "chris@twotalesanimation.com",
urgent: false,
});
expect(parsed.success).toBe(true);
});
it("defaults the stage so the add-on need not send it", () => {
const parsed = blenderManifestSchema.parse(addonManifest);
expect(parsed.stage).toBe("BLENDER");
});
it("rejects a backwards frame range rather than queueing a no-op render", () => {
const parsed = blenderManifestSchema.safeParse({ ...addonManifest, frameEnd: 900 });
expect(parsed.success).toBe(false);
});
it("accepts EEVEE, which is what the add-on reports on a non-Cycles scene", () => {
expect(blenderManifestSchema.safeParse({ ...addonManifest, engine: "BLENDER_EEVEE_NEXT" }).success).toBe(true);
});
});
+189
View File
@@ -0,0 +1,189 @@
import { Prisma, RenderJobStatus } from "@prisma/client";
import { z } from "zod";
import { db } from "@/lib/db";
import { PipelineError } from "./errors";
import { getConfigNumber } from "./config";
/**
* Standalone Blender renders.
*
* Deliberately the simplest thing in the pipeline: a queue entry and nothing
* more. No Export, no shot version, no validation, QC or delivery — the .blend
* is already set up to render, and this is "Ctrl+F12, but on the farm". Every
* lifecycle call it needs (claim, progress, fail, complete) already tolerates a
* job with no Export behind it.
*/
export const blenderManifestSchema = z.object({
stage: z.literal("BLENDER").default("BLENDER"),
blendFile: z.string().min(1),
/** Blank renders the .blend's active scene, which is usually what is wanted. */
scene: z.string().max(200).optional().nullable(),
frameStart: z.number().int(),
frameEnd: z.number().int(),
/**
* Read from the .blend at submission and passed through only so the queue can
* show where frames will land. Left unset, the file's own output path is used
* — the file is expected to be ready to render.
*/
outputPath: z.string().max(1000).optional().nullable(),
/** Overrides, all optional: an unset value leaves the .blend's own setting. */
engine: z.enum(["CYCLES", "BLENDER_EEVEE_NEXT", "BLENDER_EEVEE", "BLENDER_WORKBENCH"]).optional().nullable(),
device: z.enum(["CPU", "GPU"]).optional().nullable(),
samples: z.number().int().positive().max(100000).optional().nullable(),
resolutionPercent: z.number().int().min(1).max(100).optional().nullable(),
/** Blender version this was authored in, for diagnosing a farm-vs-desk difference. */
blenderVersion: z.string().max(50).optional().nullable(),
}).refine((m) => m.frameEnd >= m.frameStart, {
message: "frameEnd must be >= frameStart",
path: ["frameEnd"],
});
export type BlenderManifest = z.infer<typeof blenderManifestSchema>;
export const createBlenderJobSchema = z.object({
projectCode: z.string().min(1),
label: z.string().min(1).max(200),
manifest: blenderManifestSchema,
submittedByEmail: z.string().max(200).optional().nullable(),
urgent: z.boolean().optional(),
});
export type CreateBlenderJobInput = z.infer<typeof createBlenderJobSchema>;
export async function createBlenderJob(input: CreateBlenderJobInput) {
const project = await db.project.findUnique({
where: { code: input.projectCode },
select: { id: true, code: true, name: true },
});
if (!project) throw new PipelineError(404, `Project not found: ${input.projectCode}`);
const [maxAttempts, urgentThreshold] = await Promise.all([
getConfigNumber("render.maxAttempts"),
getConfigNumber("render.urgentPriorityThreshold"),
]);
const job = await db.renderJob.create({
data: {
type: "BLENDER_RENDER",
projectId: project.id,
label: input.label.trim(),
submittedBy: input.submittedByEmail?.trim() || null,
attempt: 1,
maxAttempts,
priority: input.urgent ? urgentThreshold : 50,
manifest: input.manifest as unknown as Prisma.InputJsonValue,
},
});
return {
job: {
id: job.id,
label: job.label,
status: job.status,
priority: job.priority,
frames: input.manifest.frameEnd - input.manifest.frameStart + 1,
},
project: { code: project.code, name: project.name },
};
}
export interface BlenderJobFilter {
projectId?: string;
status?: RenderJobStatus;
limit?: number;
}
export async function listBlenderJobs(filter: BlenderJobFilter = {}) {
const jobs = await db.renderJob.findMany({
where: {
type: "BLENDER_RENDER",
...(filter.projectId ? { projectId: filter.projectId } : {}),
...(filter.status ? { status: filter.status } : {}),
},
orderBy: [{ createdAt: "desc" }],
take: Math.min(200, Math.max(1, filter.limit ?? 50)),
include: { machine: { select: { name: true } } },
});
const projectIds = [...new Set(jobs.map((j) => j.projectId).filter((id): id is string => !!id))];
const projects = projectIds.length
? await db.project.findMany({ where: { id: { in: projectIds } }, select: { id: true, code: true, name: true } })
: [];
const byId = new Map(projects.map((p) => [p.id, p]));
return {
jobs: jobs.map((j) => {
const m = j.manifest as unknown as Partial<BlenderManifest>;
return {
id: j.id,
label: j.label,
status: j.status,
attempt: j.attempt,
maxAttempts: j.maxAttempts,
priority: j.priority,
progress: j.progress,
currentFrame: j.currentFrame,
totalFrames: j.totalFrames,
etaSeconds: j.etaSeconds,
errorMessage: j.errorMessage,
machineName: j.machine?.name ?? null,
submittedBy: j.submittedBy,
createdAt: j.createdAt,
startedAt: j.startedAt,
finishedAt: j.finishedAt,
projectCode: j.projectId ? byId.get(j.projectId)?.code ?? null : null,
blendFile: m.blendFile ?? null,
outputPath: m.outputPath ?? null,
frameStart: m.frameStart ?? null,
frameEnd: m.frameEnd ?? null,
};
}),
};
}
/** Cancel a queued or running Blender job. The worker learns via its progress report. */
export async function cancelBlenderJob(jobId: string, actorId?: string) {
const job = await db.renderJob.findUnique({ where: { id: jobId }, select: { id: true, type: true, status: true } });
if (!job) throw new PipelineError(404, "Render job not found");
if (job.type !== "BLENDER_RENDER") throw new PipelineError(422, "Not a Blender job");
if (!["QUEUED", "CLAIMED", "RUNNING"].includes(job.status)) {
throw new PipelineError(409, `Job is ${job.status} — only a queued or running job can be cancelled`);
}
const updated = await db.renderJob.update({
where: { id: job.id },
data: {
status: "CANCELLED",
errorMessage: `Cancelled by user${actorId ? ` ${actorId}` : ""}`,
// A claimed job keeps finishedAt null until the worker acknowledges, so a
// render still winding down is not reported as already stopped.
...(job.status === "QUEUED" ? { finishedAt: new Date() } : {}),
},
});
return { job: { id: updated.id, status: updated.status } };
}
/** Requeue a failed Blender job as a fresh attempt, reusing its manifest. */
export async function retryBlenderJob(jobId: string) {
const job = await db.renderJob.findUnique({ where: { id: jobId } });
if (!job) throw new PipelineError(404, "Render job not found");
if (job.type !== "BLENDER_RENDER") throw new PipelineError(422, "Not a Blender job");
if (!["FAILED", "CANCELLED", "EXPIRED"].includes(job.status)) {
throw new PipelineError(409, `Job is ${job.status} — only a finished-unsuccessfully job can be retried`);
}
const next = await db.renderJob.create({
data: {
type: "BLENDER_RENDER",
projectId: job.projectId,
label: job.label,
submittedBy: job.submittedBy,
attempt: job.attempt + 1,
maxAttempts: job.maxAttempts,
priority: job.priority,
manifest: job.manifest as Prisma.InputJsonValue,
},
});
return { job: { id: next.id, attempt: next.attempt, status: next.status } };
}
+36 -4
View File
@@ -3,11 +3,11 @@ import { db } from "@/lib/db";
import { PipelineError } from "./errors"; import { PipelineError } from "./errors";
import { applyTransition } from "./transitions"; import { applyTransition } from "./transitions";
import { getConfigBoolean, getConfigNumber } from "./config"; import { getConfigBoolean, getConfigNumber } from "./config";
import { machineCanClaim } from "./machines"; import { machineCanClaim, machineSupportedTypes } from "./machines";
import { reapExpiredLeases } from "./reaper"; import { reapExpiredLeases } from "./reaper";
import { buildPreviewManifest, createPreviewJob, inheritPreviewMedia } from "./preview"; import { buildPreviewManifest, createPreviewJob, inheritPreviewMedia } from "./preview";
const JOB_TYPES: RenderJobType[] = ["AE_RENDER", "PREVIEW_ONLY", "DELIVERY_BUILD"]; const JOB_TYPES: RenderJobType[] = ["AE_RENDER", "PREVIEW_ONLY", "DELIVERY_BUILD", "BLENDER_RENDER"];
/** /**
* E8 — atomically claim the next queued job (§6.4). `FOR UPDATE SKIP LOCKED` * E8 — atomically claim the next queued job (§6.4). `FOR UPDATE SKIP LOCKED`
@@ -22,10 +22,14 @@ export async function claimNextJob(machineId: string, types?: string[]) {
// Defensive sweep on each claim (studio decision 18.1-Q8) // Defensive sweep on each claim (studio decision 18.1-Q8)
await reapExpiredLeases().catch((err) => console.error("[render-pipeline] claim-time reap failed", err)); await reapExpiredLeases().catch((err) => console.error("[render-pipeline] claim-time reap failed", err));
const claimTypes = (types?.length ? types : ["AE_RENDER", "PREVIEW_ONLY", "DELIVERY_BUILD"]).filter( const requested = (types?.length ? types : ["AE_RENDER", "PREVIEW_ONLY", "DELIVERY_BUILD"]).filter(
(t): t is RenderJobType => (JOB_TYPES as string[]).includes(t) (t): t is RenderJobType => (JOB_TYPES as string[]).includes(t)
); );
if (claimTypes.length === 0) throw new PipelineError(422, "No valid job types requested"); // Enforced here rather than trusted to the worker: a machine without the tool
// installed must not be able to claim work it can only fail.
const claimTypes = machineSupportedTypes(machine, requested) as RenderJobType[];
if (requested.length === 0) throw new PipelineError(422, "No valid job types requested");
if (claimTypes.length === 0) return null;
const gate = machineCanClaim(machine, new Date()); const gate = machineCanClaim(machine, new Date());
if (gate === "none") return null; if (gate === "none") return null;
@@ -91,6 +95,7 @@ export async function claimNextJob(machineId: string, types?: string[]) {
priority: job.priority, priority: job.priority,
exportId: job.exportId, exportId: job.exportId,
deliveryId: job.deliveryId, deliveryId: job.deliveryId,
label: job.label,
leaseExpiresAt: job.leaseExpiresAt, leaseExpiresAt: job.leaseExpiresAt,
manifest: job.manifest, manifest: job.manifest,
}, },
@@ -189,6 +194,33 @@ export async function reportFail(
}); });
if (!job.export) { if (!job.export) {
// A standalone job (a Blender render) has no Export to move, but a
// transient failure should still be retried like any other work.
// Delivery builds are deliberately excluded: a half-written package
// folder wants a human to look at it before it is rebuilt.
const retryableStandalone =
!job.deliveryId && body.retryable !== false && job.attempt < job.maxAttempts;
if (retryableStandalone) {
const next = await tx.renderJob.create({
data: {
type: job.type,
projectId: job.projectId,
label: job.label,
submittedBy: job.submittedBy,
deliveryId: job.deliveryId,
attempt: job.attempt + 1,
maxAttempts: job.maxAttempts,
priority: job.priority,
manifest: job.manifest as Prisma.InputJsonValue,
},
});
return {
job: { id: job.id, status: "FAILED" },
export: null,
autoRequeued: true,
nextJob: { id: next.id, attempt: next.attempt },
};
}
return { job: { id: job.id, status: "FAILED" }, export: null, autoRequeued: false }; return { job: { id: job.id, status: "FAILED" }, export: null, autoRequeued: false };
} }
+13
View File
@@ -59,6 +59,19 @@ export function isInWindow(windows: AvailabilityWindow[], now: Date): boolean {
export type ClaimGate = "normal" | "urgent-only" | "none"; export type ClaimGate = "normal" | "urgent-only" | "none";
/**
* Job types a machine is equipped for. Capabilities are reported at
* registration; a machine without Blender installed must never be handed a
* Blender job, or it claims work it can only fail — repeatedly, until the
* attempts run out — while a machine that could have rendered it sits idle.
*/
export function machineSupportedTypes(machine: { capabilities: unknown }, requested: string[]): string[] {
const tools = ((machine.capabilities as { tools?: Record<string, unknown> } | null)?.tools ?? {}) as Record<string, unknown>;
const hasBlender = Boolean(tools.blender);
return requested.filter((type) => (type === "BLENDER_RENDER" ? hasBlender : true));
}
export function machineCanClaim(machine: Machine, now: Date): ClaimGate { export function machineCanClaim(machine: Machine, now: Date): ClaimGate {
if (!machine.enabled) return "none"; if (!machine.enabled) return "none";
if (machine.renderNowUntil && machine.renderNowUntil > now) return "normal"; if (machine.renderNowUntil && machine.renderNowUntil > now) return "normal";
@@ -0,0 +1,12 @@
-- Standalone Blender renders: a queue entry with no Export behind it.
-- AlterEnum
ALTER TYPE "RenderJobType" ADD VALUE 'BLENDER_RENDER';
-- AlterTable
ALTER TABLE "render_jobs" ADD COLUMN "projectId" TEXT;
ALTER TABLE "render_jobs" ADD COLUMN "label" TEXT;
ALTER TABLE "render_jobs" ADD COLUMN "submittedBy" TEXT;
-- CreateIndex
CREATE INDEX "render_jobs_projectId_status_idx" ON "render_jobs"("projectId", "status");
+12
View File
@@ -149,6 +149,10 @@ enum RenderJobType {
AE_RENDER AE_RENDER
PREVIEW_ONLY PREVIEW_ONLY
DELIVERY_BUILD DELIVERY_BUILD
/// A standalone Blender render: a queue entry and nothing more. It has no
/// Export, takes no shot version, and is not validated, QC'd or delivered —
/// it is "Ctrl+F12, but on the farm".
BLENDER_RENDER
} }
enum ValidationStatus { enum ValidationStatus {
@@ -915,6 +919,13 @@ model RenderJob {
exportId String? // set for AE_RENDER / PREVIEW_ONLY exportId String? // set for AE_RENDER / PREVIEW_ONLY
export Export? @relation(fields: [exportId], references: [id]) export Export? @relation(fields: [exportId], references: [id])
deliveryId String? // set for DELIVERY_BUILD (Phase 5) deliveryId String? // set for DELIVERY_BUILD (Phase 5)
/// Set for jobs that belong to a project but not to a shot — a standalone
/// Blender render. Lets the queue be filtered by production without
/// inventing an Export for work that has no shot.
projectId String?
/// Human label for such a job, since there is no shot code to show.
label String?
submittedBy String? // email/name of whoever queued it
attempt Int @default(1) attempt Int @default(1)
maxAttempts Int @default(3) maxAttempts Int @default(3)
status RenderJobStatus @default(QUEUED) status RenderJobStatus @default(QUEUED)
@@ -942,6 +953,7 @@ model RenderJob {
@@index([status, priority, createdAt]) @@index([status, priority, createdAt])
@@index([machineId, status]) @@index([machineId, status])
@@index([projectId, status])
@@map("render_jobs") @@map("render_jobs")
} }