feat(pipeline): standalone Blender renders on the same queue
Deploy / deploy (push) Successful in 7m2s
Deploy / deploy (push) Successful in 7m2s
Blender work is not a shot submission: no Export, no version, no validation, QC or delivery. It is "Ctrl+F12, but on the farm", so it is modelled as what it is — a queue entry and nothing more. That turned out to need very little. Claim, progress, fail and complete-render already tolerated a job with no Export behind them, so a Blender job reuses the whole lifecycle; RenderJob gains projectId, label and submittedBy so a job with no shot code still has something to show. Capability gating is new and was already missing: the claim now filters job types by the machine's reported tools. Without it a workstation without Blender would claim Blender work and fail it repeatedly, burning the attempts while a machine that could have rendered it sat idle. The worker asks for Blender jobs unconditionally and the server refuses them — trusting the worker to self-select would have left the same hole. The runner treats a clean exit with no saved frames as a failure: Blender exits 0 when a frame range is empty or an output path is unwritable, and a job that silently rendered nothing is the worst outcome available. Standalone jobs auto-requeue on a transient failure like any other work; delivery builds are deliberately excluded, since a half-written package wants a human first. Adds the Blender add-on (BlenderPanel/) and its docs. The add-on refuses to submit an unsaved file — the farm opens the .blend from disk, so unsaved changes would render the previous state without warning. Its payload is pinned by a contract test, since add-on and server only ever meet over JSON. Worker 1.5.0. Adds migration 20260815000000_blender_render_jobs. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,88 @@
|
|||||||
|
# VFXReview — Blender Submit
|
||||||
|
|
||||||
|
A Blender add-on that queues the open `.blend` on the studio render farm instead
|
||||||
|
of rendering it locally. Ctrl+F12, but on the farm.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## What a Blender job is (and is not)
|
||||||
|
|
||||||
|
Deliberately the simplest thing in the pipeline: **a queue entry and nothing
|
||||||
|
more**. It has no Export, takes no shot version, and is never validated, QC'd or
|
||||||
|
delivered. The `.blend` is expected to be ready to render — output path, format
|
||||||
|
and frame range already set — and the farm just runs it.
|
||||||
|
|
||||||
|
That is the whole difference from an After Effects export, which is a shot
|
||||||
|
submission and carries the full lifecycle behind it.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Install
|
||||||
|
|
||||||
|
1. **Edit → Preferences → Add-ons → Install…** and pick `vfxreview_submit.py`.
|
||||||
|
2. Enable **Render: VFXReview — Submit Render**.
|
||||||
|
3. Expand its preferences and set:
|
||||||
|
|
||||||
|
| Setting | Value |
|
||||||
|
|---|---|
|
||||||
|
| **Server URL** | `https://review.twotalesvfx.com` |
|
||||||
|
| **API Key** | matches `API_SECRET_KEY` on the server (same key the AE panel uses) |
|
||||||
|
| **Artist Email** | recorded against the job so the queue shows who submitted it |
|
||||||
|
|
||||||
|
The panel appears under **Properties → Output → VFXReview**.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Submitting
|
||||||
|
|
||||||
|
1. **Refresh Projects** (the circular arrow) and pick the project.
|
||||||
|
2. Optionally set a **Label** — defaults to the `.blend` filename, and is what
|
||||||
|
the render queue shows, since there is no shot code.
|
||||||
|
3. **Urgent** makes the job claimable outside the render window, ahead of normal
|
||||||
|
work.
|
||||||
|
4. **Submit to Render Farm**.
|
||||||
|
|
||||||
|
The panel shows the frame range, engine and output path it will send, read from
|
||||||
|
the file as saved. Nothing is changed on submission.
|
||||||
|
|
||||||
|
**The file must be saved first.** The farm opens the `.blend` from disk, so
|
||||||
|
unsaved changes would render the previous state without warning — the add-on
|
||||||
|
refuses to submit a dirty file rather than let that happen.
|
||||||
|
|
||||||
|
Track the job on the web app under **Pipeline → Blender renders**: live
|
||||||
|
progress, current frame, ETA and the machine running it, with cancel and requeue.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Which machines can render it
|
||||||
|
|
||||||
|
Only workers with `blenderPath` set in their `config.json` and the executable
|
||||||
|
present. The worker reports Blender at registration and the server hands
|
||||||
|
`BLENDER_RENDER` jobs only to machines that have it — a machine without Blender
|
||||||
|
can never claim work it could only fail.
|
||||||
|
|
||||||
|
Verify on a worker with:
|
||||||
|
|
||||||
|
```powershell
|
||||||
|
.\VFXReviewWorker.exe --check
|
||||||
|
```
|
||||||
|
|
||||||
|
Blender is listed as optional there: a machine without it stays perfectly valid
|
||||||
|
for After Effects work.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Notes
|
||||||
|
|
||||||
|
**Paths must resolve on the render machine.** The `.blend` path and its output
|
||||||
|
path are submitted as the artist sees them; the worker applies its own
|
||||||
|
`pathMappings` before opening the file. A path on a drive letter the render
|
||||||
|
account cannot see is the most common reason a job fails immediately.
|
||||||
|
|
||||||
|
**Renders as saved.** The add-on sends the scene's own settings. The manifest
|
||||||
|
supports overrides (samples, resolution %, device, engine) for future use, but
|
||||||
|
nothing in the UI sets them today — one place for render settings is the point.
|
||||||
|
|
||||||
|
**Exit code 0 with no frames is treated as a failure.** Blender exits cleanly
|
||||||
|
when a frame range is empty or an output path is unwritable, so the worker
|
||||||
|
checks that frames were actually saved before reporting success.
|
||||||
@@ -0,0 +1,251 @@
|
|||||||
|
bl_info = {
|
||||||
|
"name": "VFXReview — Submit Render",
|
||||||
|
"author": "Two Tales",
|
||||||
|
"version": (1, 0, 0),
|
||||||
|
"blender": (3, 6, 0),
|
||||||
|
"location": "Properties > Output > VFXReview",
|
||||||
|
"description": "Queue this .blend on the studio render farm instead of rendering locally",
|
||||||
|
"category": "Render",
|
||||||
|
}
|
||||||
|
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import urllib.error
|
||||||
|
import urllib.request
|
||||||
|
|
||||||
|
import bpy
|
||||||
|
from bpy.props import BoolProperty, EnumProperty, StringProperty
|
||||||
|
from bpy.types import AddonPreferences, Operator, Panel
|
||||||
|
|
||||||
|
|
||||||
|
# ── Preferences (per artist machine, like the AE panel's Settings) ────────────
|
||||||
|
|
||||||
|
class VFXRPreferences(AddonPreferences):
|
||||||
|
bl_idname = __name__
|
||||||
|
|
||||||
|
server_url: StringProperty(
|
||||||
|
name="Server URL",
|
||||||
|
default="https://review.twotalesvfx.com",
|
||||||
|
)
|
||||||
|
api_key: StringProperty(
|
||||||
|
name="API Key",
|
||||||
|
subtype="PASSWORD",
|
||||||
|
description="Matches API_SECRET_KEY on the server",
|
||||||
|
)
|
||||||
|
artist_email: StringProperty(
|
||||||
|
name="Artist Email",
|
||||||
|
description="Recorded against the job so the queue shows who submitted it",
|
||||||
|
)
|
||||||
|
|
||||||
|
def draw(self, context):
|
||||||
|
col = self.layout.column()
|
||||||
|
col.prop(self, "server_url")
|
||||||
|
col.prop(self, "api_key")
|
||||||
|
col.prop(self, "artist_email")
|
||||||
|
|
||||||
|
|
||||||
|
def prefs():
|
||||||
|
return bpy.context.preferences.addons[__name__].preferences
|
||||||
|
|
||||||
|
|
||||||
|
# ── API ───────────────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
def api_request(path, payload=None, method="GET"):
|
||||||
|
"""Minimal JSON client. urllib keeps the add-on dependency-free."""
|
||||||
|
p = prefs()
|
||||||
|
if not p.server_url or not p.api_key:
|
||||||
|
raise RuntimeError("Set the server URL and API key in Add-on Preferences")
|
||||||
|
|
||||||
|
url = p.server_url.rstrip("/") + path
|
||||||
|
data = json.dumps(payload).encode("utf-8") if payload is not None else None
|
||||||
|
req = urllib.request.Request(url, data=data, method=method)
|
||||||
|
req.add_header("Authorization", "Bearer " + p.api_key)
|
||||||
|
req.add_header("Accept", "application/json")
|
||||||
|
if data:
|
||||||
|
req.add_header("Content-Type", "application/json")
|
||||||
|
|
||||||
|
try:
|
||||||
|
with urllib.request.urlopen(req, timeout=30) as res:
|
||||||
|
return json.loads(res.read().decode("utf-8"))
|
||||||
|
except urllib.error.HTTPError as e:
|
||||||
|
body = e.read().decode("utf-8", "replace")
|
||||||
|
try:
|
||||||
|
message = json.loads(body).get("error", body)
|
||||||
|
except Exception:
|
||||||
|
message = body
|
||||||
|
raise RuntimeError("Server said %s: %s" % (e.code, message))
|
||||||
|
except urllib.error.URLError as e:
|
||||||
|
raise RuntimeError("Could not reach %s (%s)" % (url, e.reason))
|
||||||
|
|
||||||
|
|
||||||
|
_project_cache = []
|
||||||
|
|
||||||
|
|
||||||
|
def project_items(self, context):
|
||||||
|
"""Project dropdown. Cached because Blender calls this on every redraw."""
|
||||||
|
if not _project_cache:
|
||||||
|
return [("", "Refresh projects…", "")]
|
||||||
|
return [(p["code"], "%s — %s" % (p["code"], p["name"]), "") for p in _project_cache]
|
||||||
|
|
||||||
|
|
||||||
|
# ── Scene → job ───────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
def scene_summary(scene):
|
||||||
|
"""
|
||||||
|
Everything the farm needs, read from the file as saved. The .blend is
|
||||||
|
expected to be ready to render, so nothing here changes settings — it only
|
||||||
|
reports them.
|
||||||
|
"""
|
||||||
|
return {
|
||||||
|
"blendFile": bpy.data.filepath.replace("\\", "/"),
|
||||||
|
"scene": scene.name,
|
||||||
|
"frameStart": scene.frame_start,
|
||||||
|
"frameEnd": scene.frame_end,
|
||||||
|
"outputPath": bpy.path.abspath(scene.render.filepath).replace("\\", "/"),
|
||||||
|
"engine": scene.render.engine,
|
||||||
|
"blenderVersion": bpy.app.version_string,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
class VFXR_OT_refresh_projects(Operator):
|
||||||
|
bl_idname = "vfxreview.refresh_projects"
|
||||||
|
bl_label = "Refresh Projects"
|
||||||
|
bl_description = "Load the project list from the server"
|
||||||
|
|
||||||
|
def execute(self, context):
|
||||||
|
global _project_cache
|
||||||
|
try:
|
||||||
|
data = api_request("/api/ext/projects")
|
||||||
|
except RuntimeError as err:
|
||||||
|
self.report({"ERROR"}, str(err))
|
||||||
|
return {"CANCELLED"}
|
||||||
|
|
||||||
|
_project_cache = data.get("projects", [])
|
||||||
|
self.report({"INFO"}, "%d project(s) loaded" % len(_project_cache))
|
||||||
|
return {"FINISHED"}
|
||||||
|
|
||||||
|
|
||||||
|
class VFXR_OT_submit(Operator):
|
||||||
|
bl_idname = "vfxreview.submit"
|
||||||
|
bl_label = "Submit to Render Farm"
|
||||||
|
bl_description = "Queue this .blend on the farm instead of rendering locally"
|
||||||
|
|
||||||
|
def execute(self, context):
|
||||||
|
scene = context.scene
|
||||||
|
settings = scene.vfxreview
|
||||||
|
|
||||||
|
if not bpy.data.filepath:
|
||||||
|
self.report({"ERROR"}, "Save the .blend first — the farm opens it from disk")
|
||||||
|
return {"CANCELLED"}
|
||||||
|
if bpy.data.is_dirty:
|
||||||
|
# Unsaved changes would render the previous state without warning.
|
||||||
|
self.report({"ERROR"}, "Unsaved changes — save the file before submitting")
|
||||||
|
return {"CANCELLED"}
|
||||||
|
if not settings.project_code:
|
||||||
|
self.report({"ERROR"}, "Pick a project (Refresh Projects first)")
|
||||||
|
return {"CANCELLED"}
|
||||||
|
|
||||||
|
manifest = scene_summary(scene)
|
||||||
|
if manifest["frameEnd"] < manifest["frameStart"]:
|
||||||
|
self.report({"ERROR"}, "Frame end is before frame start")
|
||||||
|
return {"CANCELLED"}
|
||||||
|
|
||||||
|
label = settings.label.strip() or os.path.splitext(os.path.basename(bpy.data.filepath))[0]
|
||||||
|
|
||||||
|
try:
|
||||||
|
result = api_request(
|
||||||
|
"/api/ext/render/blender",
|
||||||
|
{
|
||||||
|
"projectCode": settings.project_code,
|
||||||
|
"label": label,
|
||||||
|
"manifest": manifest,
|
||||||
|
"submittedByEmail": prefs().artist_email or None,
|
||||||
|
"urgent": settings.urgent,
|
||||||
|
},
|
||||||
|
method="POST",
|
||||||
|
)
|
||||||
|
except RuntimeError as err:
|
||||||
|
self.report({"ERROR"}, str(err))
|
||||||
|
return {"CANCELLED"}
|
||||||
|
|
||||||
|
job = result.get("job", {})
|
||||||
|
self.report(
|
||||||
|
{"INFO"},
|
||||||
|
"Queued '%s' — %d frames%s" % (label, job.get("frames", 0), " (urgent)" if settings.urgent else ""),
|
||||||
|
)
|
||||||
|
return {"FINISHED"}
|
||||||
|
|
||||||
|
|
||||||
|
class VFXRSceneSettings(bpy.types.PropertyGroup):
|
||||||
|
project_code: EnumProperty(name="Project", items=project_items)
|
||||||
|
label: StringProperty(
|
||||||
|
name="Label",
|
||||||
|
description="Shown in the render queue. Defaults to the .blend filename",
|
||||||
|
)
|
||||||
|
urgent: BoolProperty(
|
||||||
|
name="Urgent",
|
||||||
|
description="Claimable outside the render window, ahead of normal jobs",
|
||||||
|
default=False,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
# ── UI ────────────────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
class VFXR_PT_submit(Panel):
|
||||||
|
bl_label = "VFXReview"
|
||||||
|
bl_space_type = "PROPERTIES"
|
||||||
|
bl_region_type = "WINDOW"
|
||||||
|
bl_context = "output"
|
||||||
|
|
||||||
|
def draw(self, context):
|
||||||
|
scene = context.scene
|
||||||
|
settings = scene.vfxreview
|
||||||
|
col = self.layout.column()
|
||||||
|
|
||||||
|
row = col.row(align=True)
|
||||||
|
row.prop(settings, "project_code", text="")
|
||||||
|
row.operator("vfxreview.refresh_projects", text="", icon="FILE_REFRESH")
|
||||||
|
|
||||||
|
col.prop(settings, "label")
|
||||||
|
col.prop(settings, "urgent")
|
||||||
|
|
||||||
|
box = col.box()
|
||||||
|
box.label(text="Renders as saved:", icon="INFO")
|
||||||
|
box.label(text="Frames %d–%d (%d)" % (
|
||||||
|
scene.frame_start, scene.frame_end, scene.frame_end - scene.frame_start + 1))
|
||||||
|
box.label(text="Engine: %s" % scene.render.engine)
|
||||||
|
output = bpy.path.abspath(scene.render.filepath)
|
||||||
|
box.label(text="Output: %s" % (output[-48:] if len(output) > 48 else output))
|
||||||
|
|
||||||
|
if bpy.data.is_dirty or not bpy.data.filepath:
|
||||||
|
warn = col.row()
|
||||||
|
warn.alert = True
|
||||||
|
warn.label(text="Save the file before submitting", icon="ERROR")
|
||||||
|
|
||||||
|
col.separator()
|
||||||
|
col.operator("vfxreview.submit", icon="RENDER_ANIMATION")
|
||||||
|
|
||||||
|
|
||||||
|
classes = (
|
||||||
|
VFXRPreferences,
|
||||||
|
VFXRSceneSettings,
|
||||||
|
VFXR_OT_refresh_projects,
|
||||||
|
VFXR_OT_submit,
|
||||||
|
VFXR_PT_submit,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def register():
|
||||||
|
for cls in classes:
|
||||||
|
bpy.utils.register_class(cls)
|
||||||
|
bpy.types.Scene.vfxreview = bpy.props.PointerProperty(type=VFXRSceneSettings)
|
||||||
|
|
||||||
|
|
||||||
|
def unregister():
|
||||||
|
del bpy.types.Scene.vfxreview
|
||||||
|
for cls in reversed(classes):
|
||||||
|
bpy.utils.unregister_class(cls)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
register()
|
||||||
@@ -174,6 +174,7 @@ prisma/ — schema.prisma, seed.ts
|
|||||||
| [DELIVERY_WORKFLOW.md](DELIVERY_WORKFLOW.md) | Building and shipping delivery packages (how-to) |
|
| [DELIVERY_WORKFLOW.md](DELIVERY_WORKFLOW.md) | Building and shipping delivery packages (how-to) |
|
||||||
| [VFXReviewConnector.md](VFXReviewConnector.md) | After Effects panel — setup, Shot Builder, QC, diagnostics |
|
| [VFXReviewConnector.md](VFXReviewConnector.md) | After Effects panel — setup, Shot Builder, QC, diagnostics |
|
||||||
| [RenderWorker/README.md](RenderWorker/README.md) | Render worker service — packaging, install, validation, delivery builds |
|
| [RenderWorker/README.md](RenderWorker/README.md) | Render worker service — packaging, install, validation, delivery builds |
|
||||||
|
| [BlenderPanel/README.md](BlenderPanel/README.md) | Blender add-on — queueing a .blend on the farm |
|
||||||
| [EXT_API_REFERENCE.md](EXT_API_REFERENCE.md) | External API (`/api/ext/*`) used by the panel and workers |
|
| [EXT_API_REFERENCE.md](EXT_API_REFERENCE.md) | External API (`/api/ext/*`) used by the panel and workers |
|
||||||
| [Shot task workflow.md](Shot%20task%20workflow.md) | Shot and task lifecycle in the web app |
|
| [Shot task workflow.md](Shot%20task%20workflow.md) | Shot and task lifecycle in the web app |
|
||||||
|
|
||||||
|
|||||||
@@ -89,6 +89,23 @@ Server-side switches (SystemConfig, no worker redeploy): `validation.enabled`,
|
|||||||
`validation.sampleEvery`, `validation.handleFrames`, and `validation.overrides` for per-check
|
`validation.sampleEvery`, `validation.handleFrames`, and `validation.overrides` for per-check
|
||||||
severity, e.g. `{"exr_type":"warn"}`.
|
severity, e.g. `{"exr_type":"warn"}`.
|
||||||
|
|
||||||
|
## Blender renders
|
||||||
|
|
||||||
|
The worker also claims `BLENDER_RENDER` jobs: standalone renders submitted from
|
||||||
|
the Blender add-on, with no Export behind them — nothing to validate, preview or
|
||||||
|
QC. It runs `blender -b file.blend -s N -e M -a`, parses `Fra:`/`Saved:` output
|
||||||
|
for progress, and reports complete or failed like any other job.
|
||||||
|
|
||||||
|
Set `blenderPath` in `config.json` to make a machine eligible. The version is
|
||||||
|
reported at registration and **the server only hands Blender jobs to machines
|
||||||
|
that have it**, so a workstation without Blender can never claim work it could
|
||||||
|
only fail. A machine without it stays perfectly valid for After Effects work.
|
||||||
|
|
||||||
|
Blender exits 0 when a frame range is empty or an output path is unwritable, so
|
||||||
|
a run that saves no frames is reported as a failure rather than a success.
|
||||||
|
|
||||||
|
See [BlenderPanel/README.md](../BlenderPanel/README.md).
|
||||||
|
|
||||||
## Delivery builds
|
## Delivery builds
|
||||||
|
|
||||||
The same service also claims `DELIVERY_BUILD` jobs: creating a dated package
|
The same service also claims `DELIVERY_BUILD` jobs: creating a dated package
|
||||||
@@ -174,6 +191,7 @@ Create `C:\ProgramData\VFXReviewWorker\config.json` (§7.8):
|
|||||||
"aerenderPath": "C:\\Program Files\\Adobe\\Adobe After Effects 2026\\Support Files\\aerender.exe",
|
"aerenderPath": "C:\\Program Files\\Adobe\\Adobe After Effects 2026\\Support Files\\aerender.exe",
|
||||||
"aeVersion": "24.3",
|
"aeVersion": "24.3",
|
||||||
"ffmpegPath": "C:\\pipeline\\bin\\ffmpeg.exe",
|
"ffmpegPath": "C:\\pipeline\\bin\\ffmpeg.exe",
|
||||||
|
"blenderPath": "C:\\Program Files\\Blender Foundation\\Blender 4.2\\blender.exe",
|
||||||
"plateSearchRoots": ["V:/_FOOTAGE/UNG"],
|
"plateSearchRoots": ["V:/_FOOTAGE/UNG"],
|
||||||
"pathMappings": [
|
"pathMappings": [
|
||||||
{ "from": "//SAN/", "to": "S:/" }
|
{ "from": "//SAN/", "to": "S:/" }
|
||||||
|
|||||||
@@ -37,7 +37,20 @@ public sealed class ApiClient
|
|||||||
hostname = Dns.GetHostName(),
|
hostname = Dns.GetHostName(),
|
||||||
workerVersion = o.WorkerVersion,
|
workerVersion = o.WorkerVersion,
|
||||||
aeVersion = o.AeVersion,
|
aeVersion = o.AeVersion,
|
||||||
capabilities = new { maxConcurrentJobs = 1, tools = new { ffmpeg = o.FfmpegPath != null, oiiotool = o.OiiotoolPath != null } },
|
// The server hands out BLENDER_RENDER work on the strength of the
|
||||||
|
// blender entry, so it reports the resolved path rather than a bare
|
||||||
|
// flag: a machine claiming Blender jobs it cannot run is worse than
|
||||||
|
// one that never claims them.
|
||||||
|
capabilities = new
|
||||||
|
{
|
||||||
|
maxConcurrentJobs = 1,
|
||||||
|
tools = new
|
||||||
|
{
|
||||||
|
ffmpeg = o.FfmpegPath != null,
|
||||||
|
oiiotool = o.OiiotoolPath != null,
|
||||||
|
blender = !string.IsNullOrWhiteSpace(o.BlenderPath) && File.Exists(o.BlenderPath) ? o.BlenderPath : null,
|
||||||
|
},
|
||||||
|
},
|
||||||
}, Json, ct);
|
}, Json, ct);
|
||||||
res.EnsureSuccessStatusCode();
|
res.EnsureSuccessStatusCode();
|
||||||
return (await res.Content.ReadFromJsonAsync<RegisterResponse>(Json, ct))!;
|
return (await res.Content.ReadFromJsonAsync<RegisterResponse>(Json, ct))!;
|
||||||
|
|||||||
@@ -0,0 +1,68 @@
|
|||||||
|
using System.Text.RegularExpressions;
|
||||||
|
|
||||||
|
namespace VFXReviewWorker;
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Parses `blender -b` stdout. Blender reports the frame it is working on in
|
||||||
|
/// its status lines ("Fra:1042 Mem:512.00M ...") and confirms each written file
|
||||||
|
/// with "Saved: '...'". The saved lines are the reliable signal — a frame can
|
||||||
|
/// be reported for a long time while it renders — so progress counts those and
|
||||||
|
/// falls back to Fra: for a first sign of life.
|
||||||
|
/// </summary>
|
||||||
|
public static partial class BlenderProgressParser
|
||||||
|
{
|
||||||
|
[GeneratedRegex(@"^Fra:(\d+)", RegexOptions.Compiled)]
|
||||||
|
private static partial Regex FrameLine();
|
||||||
|
|
||||||
|
[GeneratedRegex(@"^Saved:\s*'(.+)'", RegexOptions.Compiled)]
|
||||||
|
private static partial Regex SavedLine();
|
||||||
|
|
||||||
|
[GeneratedRegex(@"^(Error|Warning): |^Error: ", RegexOptions.Compiled | RegexOptions.IgnoreCase)]
|
||||||
|
private static partial Regex ErrorLine();
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Failures that will repeat identically on another machine — a missing
|
||||||
|
/// file, an unknown scene, an unwritable output. Retrying these burns the
|
||||||
|
/// queue for nothing; a human has to fix the .blend or the path.
|
||||||
|
/// </summary>
|
||||||
|
private static readonly string[] NonRetryablePatterns =
|
||||||
|
{
|
||||||
|
"cannot read file",
|
||||||
|
"file not found",
|
||||||
|
"unable to open",
|
||||||
|
"is not a blend file",
|
||||||
|
"failed to read blend file",
|
||||||
|
"unknown scene",
|
||||||
|
"scene not found",
|
||||||
|
"unable to create directory",
|
||||||
|
"cannot write file",
|
||||||
|
"no camera",
|
||||||
|
"python script fail",
|
||||||
|
"unknown argument",
|
||||||
|
};
|
||||||
|
|
||||||
|
/// <summary>The frame Blender is currently working on, or null.</summary>
|
||||||
|
public static int? ParseFrame(string line)
|
||||||
|
{
|
||||||
|
var m = FrameLine().Match(line);
|
||||||
|
return m.Success && int.TryParse(m.Groups[1].Value, out var f) ? f : null;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>The path of a frame Blender has just written, or null.</summary>
|
||||||
|
public static string? ParseSaved(string line)
|
||||||
|
{
|
||||||
|
var m = SavedLine().Match(line);
|
||||||
|
return m.Success ? m.Groups[1].Value : null;
|
||||||
|
}
|
||||||
|
|
||||||
|
public static bool IsErrorLine(string line) => ErrorLine().IsMatch(line);
|
||||||
|
|
||||||
|
public static bool IsNonRetryableError(string line)
|
||||||
|
{
|
||||||
|
foreach (var p in NonRetryablePatterns)
|
||||||
|
{
|
||||||
|
if (line.Contains(p, StringComparison.OrdinalIgnoreCase)) return true;
|
||||||
|
}
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,260 @@
|
|||||||
|
using System.Diagnostics;
|
||||||
|
using System.Text;
|
||||||
|
using Microsoft.Extensions.Logging;
|
||||||
|
|
||||||
|
namespace VFXReviewWorker;
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Standalone Blender renders: `blender -b file.blend -s N -e M -a`.
|
||||||
|
///
|
||||||
|
/// The .blend is expected to be ready to render — output path, format and
|
||||||
|
/// frame range already set by whoever built it. Overrides are only applied
|
||||||
|
/// when the submission asked for them, so this stays "Ctrl+F12 on the farm"
|
||||||
|
/// rather than a second place where render settings can drift.
|
||||||
|
/// </summary>
|
||||||
|
public sealed class BlenderRunner
|
||||||
|
{
|
||||||
|
private readonly WorkerOptions _options;
|
||||||
|
private readonly ApiClient _api;
|
||||||
|
private readonly ILogger<BlenderRunner> _log;
|
||||||
|
|
||||||
|
public BlenderRunner(WorkerOptions options, ApiClient api, ILogger<BlenderRunner> log)
|
||||||
|
{
|
||||||
|
_options = options;
|
||||||
|
_api = api;
|
||||||
|
_log = log;
|
||||||
|
}
|
||||||
|
|
||||||
|
public async Task<RenderResult> RunAsync(
|
||||||
|
ClaimedJob job,
|
||||||
|
BlenderManifest m,
|
||||||
|
string machineId,
|
||||||
|
ServerConfig config,
|
||||||
|
CancellationToken cancelJob,
|
||||||
|
CancellationToken shutdown)
|
||||||
|
{
|
||||||
|
var mapper = new PathMapper(_options.PathMappings);
|
||||||
|
var blendLocal = mapper.Map(m.BlendFile);
|
||||||
|
var totalFrames = m.TotalFrames;
|
||||||
|
|
||||||
|
if (!File.Exists(blendLocal))
|
||||||
|
{
|
||||||
|
return new RenderResult(false, -1, Retryable: false,
|
||||||
|
$"Blend file not found on this machine: {blendLocal}", "", "", 0);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Argument order matters to Blender: everything before -a configures the
|
||||||
|
// render, and -a must come last or it renders with the old settings.
|
||||||
|
var args = new List<string> { "-b", blendLocal };
|
||||||
|
if (!string.IsNullOrWhiteSpace(m.Scene)) { args.Add("-S"); args.Add(m.Scene!); }
|
||||||
|
if (!string.IsNullOrWhiteSpace(m.OutputPath))
|
||||||
|
{
|
||||||
|
args.Add("-o");
|
||||||
|
args.Add(mapper.Map(m.OutputPath!));
|
||||||
|
}
|
||||||
|
if (!string.IsNullOrWhiteSpace(m.Engine)) { args.Add("-E"); args.Add(m.Engine!); }
|
||||||
|
|
||||||
|
var pythonOverrides = BuildOverrideExpression(m);
|
||||||
|
if (pythonOverrides is not null) { args.Add("--python-expr"); args.Add(pythonOverrides); }
|
||||||
|
|
||||||
|
args.Add("-s"); args.Add(m.FrameStart.ToString());
|
||||||
|
args.Add("-e"); args.Add(m.FrameEnd.ToString());
|
||||||
|
args.Add("-a");
|
||||||
|
|
||||||
|
_log.LogInformation("Rendering {Label}: {Blend} frames {Start}-{End}",
|
||||||
|
job.Label ?? job.Id, blendLocal, m.FrameStart, m.FrameEnd);
|
||||||
|
|
||||||
|
return await ExecuteAsync(job.Id, machineId, args, totalFrames, m.FrameStart, config,
|
||||||
|
$"blender_{job.Id}.log", cancelJob, shutdown);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Settings Blender has no command-line flag for. Returns null when the
|
||||||
|
/// submission asked for no overrides, so the file renders exactly as saved.
|
||||||
|
/// </summary>
|
||||||
|
private static string? BuildOverrideExpression(BlenderManifest m)
|
||||||
|
{
|
||||||
|
var parts = new List<string>();
|
||||||
|
if (m.Samples is int samples)
|
||||||
|
{
|
||||||
|
parts.Add($"getattr(S.cycles, 'samples', None) and setattr(S.cycles, 'samples', {samples})");
|
||||||
|
}
|
||||||
|
if (m.ResolutionPercent is int pct)
|
||||||
|
{
|
||||||
|
parts.Add($"setattr(S.render, 'resolution_percentage', {pct})");
|
||||||
|
}
|
||||||
|
if (!string.IsNullOrWhiteSpace(m.Device))
|
||||||
|
{
|
||||||
|
var deviceValue = m.Device!.Equals("GPU", StringComparison.OrdinalIgnoreCase) ? "GPU" : "CPU";
|
||||||
|
parts.Add($"hasattr(S, 'cycles') and setattr(S.cycles, 'device', '{deviceValue}')");
|
||||||
|
}
|
||||||
|
if (parts.Count == 0) return null;
|
||||||
|
|
||||||
|
return "import bpy; S = bpy.context.scene; " + string.Join("; ", parts);
|
||||||
|
}
|
||||||
|
|
||||||
|
private async Task<RenderResult> ExecuteAsync(
|
||||||
|
string jobId,
|
||||||
|
string machineId,
|
||||||
|
List<string> args,
|
||||||
|
int totalFrames,
|
||||||
|
int frameStart,
|
||||||
|
ServerConfig config,
|
||||||
|
string logFileName,
|
||||||
|
CancellationToken cancelJob,
|
||||||
|
CancellationToken shutdown)
|
||||||
|
{
|
||||||
|
var fullLogPath = Path.Combine(WorkerOptions.LogsDir, logFileName);
|
||||||
|
Directory.CreateDirectory(WorkerOptions.LogsDir);
|
||||||
|
|
||||||
|
var psi = new ProcessStartInfo
|
||||||
|
{
|
||||||
|
FileName = _options.BlenderPath,
|
||||||
|
RedirectStandardOutput = true,
|
||||||
|
RedirectStandardError = true,
|
||||||
|
UseShellExecute = false,
|
||||||
|
CreateNoWindow = true,
|
||||||
|
StandardOutputEncoding = Encoding.UTF8,
|
||||||
|
StandardErrorEncoding = Encoding.UTF8,
|
||||||
|
};
|
||||||
|
foreach (var a in args) psi.ArgumentList.Add(a);
|
||||||
|
|
||||||
|
var tail = new LogTail(200);
|
||||||
|
var eta = new EtaCalculator();
|
||||||
|
var sw = Stopwatch.StartNew();
|
||||||
|
var lastProgressAt = DateTimeOffset.UtcNow;
|
||||||
|
var lastReportAt = DateTimeOffset.MinValue;
|
||||||
|
var framesSaved = 0;
|
||||||
|
var currentFrame = frameStart;
|
||||||
|
string? firstError = null;
|
||||||
|
var nonRetryable = false;
|
||||||
|
var cancelledByServer = false;
|
||||||
|
|
||||||
|
using var process = new Process { StartInfo = psi };
|
||||||
|
await using var fullLog = new StreamWriter(fullLogPath, append: false, Encoding.UTF8);
|
||||||
|
var logLock = new object();
|
||||||
|
|
||||||
|
void HandleLine(string? line, bool isStderr)
|
||||||
|
{
|
||||||
|
if (line is null) return;
|
||||||
|
lock (logLock)
|
||||||
|
{
|
||||||
|
fullLog.WriteLine(line);
|
||||||
|
tail.Add(line);
|
||||||
|
}
|
||||||
|
|
||||||
|
var frame = BlenderProgressParser.ParseFrame(line);
|
||||||
|
if (frame is not null)
|
||||||
|
{
|
||||||
|
// A frame line only proves Blender is alive; a long frame emits
|
||||||
|
// many of them, so this feeds the stall watchdog, not progress.
|
||||||
|
currentFrame = frame.Value;
|
||||||
|
lastProgressAt = DateTimeOffset.UtcNow;
|
||||||
|
}
|
||||||
|
if (BlenderProgressParser.ParseSaved(line) is not null)
|
||||||
|
{
|
||||||
|
framesSaved++;
|
||||||
|
lastProgressAt = DateTimeOffset.UtcNow;
|
||||||
|
eta.RecordFrame(lastProgressAt);
|
||||||
|
}
|
||||||
|
if ((isStderr || BlenderProgressParser.IsErrorLine(line)) && !string.IsNullOrWhiteSpace(line))
|
||||||
|
{
|
||||||
|
if (BlenderProgressParser.IsErrorLine(line)) firstError ??= line.Trim();
|
||||||
|
if (BlenderProgressParser.IsNonRetryableError(line)) nonRetryable = true;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
process.OutputDataReceived += (_, e) => HandleLine(e.Data, isStderr: false);
|
||||||
|
process.ErrorDataReceived += (_, e) => HandleLine(e.Data, isStderr: true);
|
||||||
|
|
||||||
|
try
|
||||||
|
{
|
||||||
|
process.Start();
|
||||||
|
}
|
||||||
|
catch (Exception ex)
|
||||||
|
{
|
||||||
|
return new RenderResult(false, -1, Retryable: false,
|
||||||
|
$"Failed to launch Blender at {_options.BlenderPath}: {ex.Message}",
|
||||||
|
tail.ToString(), fullLogPath, 0);
|
||||||
|
}
|
||||||
|
process.BeginOutputReadLine();
|
||||||
|
process.BeginErrorReadLine();
|
||||||
|
|
||||||
|
while (!process.HasExited)
|
||||||
|
{
|
||||||
|
await Task.Delay(TimeSpan.FromSeconds(2), CancellationToken.None);
|
||||||
|
|
||||||
|
if (cancelJob.IsCancellationRequested || shutdown.IsCancellationRequested)
|
||||||
|
{
|
||||||
|
_log.LogWarning("Cancellation requested — killing Blender tree for job {JobId}", jobId);
|
||||||
|
KillTree(process);
|
||||||
|
cancelledByServer = cancelJob.IsCancellationRequested;
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
|
||||||
|
if ((DateTimeOffset.UtcNow - lastProgressAt).TotalSeconds > config.StallTimeoutSeconds)
|
||||||
|
{
|
||||||
|
_log.LogError("Blender stalled ({Sec}s without output) — killing job {JobId}", config.StallTimeoutSeconds, jobId);
|
||||||
|
KillTree(process);
|
||||||
|
await WaitForExitSafe(process);
|
||||||
|
return new RenderResult(false, -2, Retryable: true,
|
||||||
|
$"Blender produced no output for {config.StallTimeoutSeconds}s (stall timeout)",
|
||||||
|
tail.ToString(), fullLogPath, sw.Elapsed.TotalSeconds);
|
||||||
|
}
|
||||||
|
|
||||||
|
if ((DateTimeOffset.UtcNow - lastReportAt).TotalSeconds >= 10)
|
||||||
|
{
|
||||||
|
lastReportAt = DateTimeOffset.UtcNow;
|
||||||
|
var progress = totalFrames > 0 ? Math.Min(1.0, (double)framesSaved / totalFrames) : 0;
|
||||||
|
var resp = await _api.ProgressAsync(jobId, machineId, progress, currentFrame, totalFrames,
|
||||||
|
eta.EstimateSeconds(totalFrames - framesSaved), tail.ToString(), shutdown);
|
||||||
|
if (resp?.CancelRequested == true)
|
||||||
|
{
|
||||||
|
_log.LogWarning("Server requested cancel for job {JobId}", jobId);
|
||||||
|
KillTree(process);
|
||||||
|
cancelledByServer = true;
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
await WaitForExitSafe(process);
|
||||||
|
sw.Stop();
|
||||||
|
lock (logLock) { fullLog.Flush(); }
|
||||||
|
|
||||||
|
if (cancelledByServer)
|
||||||
|
{
|
||||||
|
return new RenderResult(false, -3, Retryable: false, "Cancelled", tail.ToString(), fullLogPath, sw.Elapsed.TotalSeconds);
|
||||||
|
}
|
||||||
|
|
||||||
|
var exitCode = process.ExitCode;
|
||||||
|
if (exitCode != 0)
|
||||||
|
{
|
||||||
|
return new RenderResult(false, exitCode, Retryable: !nonRetryable,
|
||||||
|
firstError ?? $"Blender exited with code {exitCode}",
|
||||||
|
tail.ToString(), fullLogPath, sw.Elapsed.TotalSeconds);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Blender exits 0 having rendered nothing if the range is empty or the
|
||||||
|
// output path is unwritable, so a silent success is treated as failure.
|
||||||
|
if (framesSaved == 0)
|
||||||
|
{
|
||||||
|
return new RenderResult(false, 0, Retryable: false,
|
||||||
|
"Blender exited cleanly but saved no frames — check the output path and frame range in the .blend",
|
||||||
|
tail.ToString(), fullLogPath, sw.Elapsed.TotalSeconds);
|
||||||
|
}
|
||||||
|
|
||||||
|
_log.LogInformation("Job {JobId}: {Saved}/{Total} frames in {Sec:F0}s", jobId, framesSaved, totalFrames, sw.Elapsed.TotalSeconds);
|
||||||
|
return new RenderResult(true, 0, true, null, tail.ToString(), fullLogPath, sw.Elapsed.TotalSeconds);
|
||||||
|
}
|
||||||
|
|
||||||
|
private static void KillTree(Process process)
|
||||||
|
{
|
||||||
|
try { if (!process.HasExited) process.Kill(entireProcessTree: true); } catch { }
|
||||||
|
}
|
||||||
|
|
||||||
|
private static async Task WaitForExitSafe(Process process)
|
||||||
|
{
|
||||||
|
try { await process.WaitForExitAsync(); } catch { }
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -49,6 +49,9 @@ public sealed class ClaimedJob
|
|||||||
public int Attempt { get; set; }
|
public int Attempt { get; set; }
|
||||||
public int MaxAttempts { get; set; }
|
public int MaxAttempts { get; set; }
|
||||||
public string? ExportId { get; set; }
|
public string? ExportId { get; set; }
|
||||||
|
public string? DeliveryId { get; set; }
|
||||||
|
/// <summary>Set for jobs with no shot behind them; shown instead of a shot code.</summary>
|
||||||
|
public string? Label { get; set; }
|
||||||
public DateTimeOffset? LeaseExpiresAt { get; set; }
|
public DateTimeOffset? LeaseExpiresAt { get; set; }
|
||||||
|
|
||||||
/// <summary>Manifest shape depends on the job type — deserialize per stage.</summary>
|
/// <summary>Manifest shape depends on the job type — deserialize per stage.</summary>
|
||||||
@@ -65,6 +68,32 @@ public sealed class ClaimedJob
|
|||||||
public DeliveryManifest AsDeliveryManifest() =>
|
public DeliveryManifest AsDeliveryManifest() =>
|
||||||
Manifest.Deserialize<DeliveryManifest>(WorkerOptions.JsonOpts)
|
Manifest.Deserialize<DeliveryManifest>(WorkerOptions.JsonOpts)
|
||||||
?? throw new InvalidOperationException("Job manifest is not a delivery manifest");
|
?? throw new InvalidOperationException("Job manifest is not a delivery manifest");
|
||||||
|
|
||||||
|
public BlenderManifest AsBlenderManifest() =>
|
||||||
|
Manifest.Deserialize<BlenderManifest>(WorkerOptions.JsonOpts)
|
||||||
|
?? throw new InvalidOperationException("Job manifest is not a Blender manifest");
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// A standalone Blender render. Overrides are all optional: an unset value
|
||||||
|
/// leaves whatever the .blend was saved with, which is the point — the file is
|
||||||
|
/// expected to be ready to render.
|
||||||
|
/// </summary>
|
||||||
|
public sealed class BlenderManifest
|
||||||
|
{
|
||||||
|
public string Stage { get; set; } = "BLENDER";
|
||||||
|
public string BlendFile { get; set; } = "";
|
||||||
|
public string? Scene { get; set; }
|
||||||
|
public int FrameStart { get; set; }
|
||||||
|
public int FrameEnd { get; set; }
|
||||||
|
public string? OutputPath { get; set; }
|
||||||
|
public string? Engine { get; set; }
|
||||||
|
public string? Device { get; set; }
|
||||||
|
public int? Samples { get; set; }
|
||||||
|
public int? ResolutionPercent { get; set; }
|
||||||
|
public string? BlenderVersion { get; set; }
|
||||||
|
|
||||||
|
public int TotalFrames => FrameEnd - FrameStart + 1;
|
||||||
}
|
}
|
||||||
|
|
||||||
/// <summary>DELIVERY_BUILD manifest (server-built, RenderPipeline2 §11).</summary>
|
/// <summary>DELIVERY_BUILD manifest (server-built, RenderPipeline2 §11).</summary>
|
||||||
|
|||||||
@@ -16,11 +16,12 @@ public sealed class JobExecutor
|
|||||||
private readonly PreviewStage _preview;
|
private readonly PreviewStage _preview;
|
||||||
private readonly ValidationStage _validation;
|
private readonly ValidationStage _validation;
|
||||||
private readonly DeliveryStage _delivery;
|
private readonly DeliveryStage _delivery;
|
||||||
|
private readonly BlenderRunner _blender;
|
||||||
private readonly ReportSpooler _spooler;
|
private readonly ReportSpooler _spooler;
|
||||||
private readonly ILogger<JobExecutor> _log;
|
private readonly ILogger<JobExecutor> _log;
|
||||||
|
|
||||||
public JobExecutor(WorkerOptions options, ApiClient api, AerenderRunner runner, PreviewStage preview,
|
public JobExecutor(WorkerOptions options, ApiClient api, AerenderRunner runner, PreviewStage preview,
|
||||||
ValidationStage validation, DeliveryStage delivery, ReportSpooler spooler, ILogger<JobExecutor> log)
|
ValidationStage validation, DeliveryStage delivery, BlenderRunner blender, ReportSpooler spooler, ILogger<JobExecutor> log)
|
||||||
{
|
{
|
||||||
_options = options;
|
_options = options;
|
||||||
_api = api;
|
_api = api;
|
||||||
@@ -28,6 +29,7 @@ public sealed class JobExecutor
|
|||||||
_preview = preview;
|
_preview = preview;
|
||||||
_validation = validation;
|
_validation = validation;
|
||||||
_delivery = delivery;
|
_delivery = delivery;
|
||||||
|
_blender = blender;
|
||||||
_spooler = spooler;
|
_spooler = spooler;
|
||||||
_log = log;
|
_log = log;
|
||||||
}
|
}
|
||||||
@@ -46,6 +48,12 @@ public sealed class JobExecutor
|
|||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
if (job.Type == "BLENDER_RENDER")
|
||||||
|
{
|
||||||
|
await RunBlenderAsync(job, machineId, config, cancelJob, shutdown);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
var manifest = job.AsRenderManifest();
|
var manifest = job.AsRenderManifest();
|
||||||
var mapper = new PathMapper(_options.PathMappings);
|
var mapper = new PathMapper(_options.PathMappings);
|
||||||
var outputDirLocal = mapper.Map(manifest.OutputDir);
|
var outputDirLocal = mapper.Map(manifest.OutputDir);
|
||||||
@@ -169,6 +177,74 @@ public sealed class JobExecutor
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// BLENDER_RENDER job: a standalone render with no Export behind it, so
|
||||||
|
/// there is nothing to validate, preview or QC — it finishes at
|
||||||
|
/// complete-render and that is the whole lifecycle.
|
||||||
|
/// </summary>
|
||||||
|
private async Task RunBlenderAsync(ClaimedJob job, string machineId, ServerConfig config,
|
||||||
|
CancellationToken cancelJob, CancellationToken shutdown)
|
||||||
|
{
|
||||||
|
var manifest = job.AsBlenderManifest();
|
||||||
|
|
||||||
|
WriteState(new CurrentJobState
|
||||||
|
{
|
||||||
|
JobId = job.Id,
|
||||||
|
ExportId = null,
|
||||||
|
MachineId = machineId,
|
||||||
|
OutputDirLocal = manifest.OutputPath ?? "",
|
||||||
|
ClaimedAt = DateTimeOffset.UtcNow,
|
||||||
|
});
|
||||||
|
|
||||||
|
try
|
||||||
|
{
|
||||||
|
var result = await _blender.RunAsync(job, manifest, machineId, config, cancelJob, shutdown);
|
||||||
|
var logKey = string.IsNullOrEmpty(result.FullLogPath)
|
||||||
|
? null
|
||||||
|
: await _api.UploadLogAsync(job.Id, machineId, result.FullLogPath, shutdown);
|
||||||
|
|
||||||
|
if (result.Success)
|
||||||
|
{
|
||||||
|
_log.LogInformation("Blender job {JobId} complete: {Frames} frames in {Sec:F0}s",
|
||||||
|
job.Id, manifest.TotalFrames, result.RenderSeconds);
|
||||||
|
_spooler.Enqueue("complete-render", job.Id, new Dictionary<string, object?>
|
||||||
|
{
|
||||||
|
["machineId"] = machineId,
|
||||||
|
["renderSeconds"] = (int)result.RenderSeconds,
|
||||||
|
["logFileKey"] = logKey,
|
||||||
|
["logTail"] = result.LogTail,
|
||||||
|
});
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
_spooler.Enqueue("fail", job.Id, new Dictionary<string, object?>
|
||||||
|
{
|
||||||
|
["machineId"] = machineId,
|
||||||
|
["stage"] = "BLENDER",
|
||||||
|
["exitCode"] = result.ExitCode,
|
||||||
|
["errorMessage"] = result.ErrorMessage,
|
||||||
|
["logTail"] = result.LogTail,
|
||||||
|
["logFileKey"] = logKey,
|
||||||
|
["retryable"] = result.ErrorMessage == "Cancelled" ? false : result.Retryable,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
catch (Exception ex)
|
||||||
|
{
|
||||||
|
_log.LogError(ex, "Unexpected Blender error for job {JobId}", job.Id);
|
||||||
|
_spooler.Enqueue("fail", job.Id, new Dictionary<string, object?>
|
||||||
|
{
|
||||||
|
["machineId"] = machineId,
|
||||||
|
["stage"] = "BLENDER",
|
||||||
|
["errorMessage"] = $"Worker exception: {ex.Message}",
|
||||||
|
["retryable"] = true,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
finally
|
||||||
|
{
|
||||||
|
ClearState();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
/// <summary>
|
/// <summary>
|
||||||
/// DELIVERY_BUILD job: build the dated package folder from already-QC'd
|
/// DELIVERY_BUILD job: build the dated package folder from already-QC'd
|
||||||
/// exports. No AE involved — folders, links and verification only.
|
/// exports. No AE involved — folders, links and verification only.
|
||||||
|
|||||||
@@ -38,6 +38,11 @@ public static class Preflight
|
|||||||
Console.WriteLine();
|
Console.WriteLine();
|
||||||
|
|
||||||
ok &= Report("aerender.exe", options.AerenderPath, File.Exists(options.AerenderPath));
|
ok &= Report("aerender.exe", options.AerenderPath, File.Exists(options.AerenderPath));
|
||||||
|
// Not required: a machine without Blender simply never claims Blender
|
||||||
|
// jobs, so this reports rather than fails.
|
||||||
|
Report("blender.exe", options.BlenderPath ?? "(not configured)",
|
||||||
|
!string.IsNullOrWhiteSpace(options.BlenderPath) && File.Exists(options.BlenderPath),
|
||||||
|
"needed only for Blender render jobs");
|
||||||
var afterFx = options.ResolveAfterFxPath();
|
var afterFx = options.ResolveAfterFxPath();
|
||||||
ok &= Report("AfterFX.com", afterFx, File.Exists(afterFx),
|
ok &= Report("AfterFX.com", afterFx, File.Exists(afterFx),
|
||||||
"needed for preview builds only");
|
"needed for preview builds only");
|
||||||
|
|||||||
@@ -41,6 +41,7 @@ builder.Services.AddSingleton<AerenderRunner>();
|
|||||||
builder.Services.AddSingleton<PreviewStage>();
|
builder.Services.AddSingleton<PreviewStage>();
|
||||||
builder.Services.AddSingleton<ValidationStage>();
|
builder.Services.AddSingleton<ValidationStage>();
|
||||||
builder.Services.AddSingleton<DeliveryStage>();
|
builder.Services.AddSingleton<DeliveryStage>();
|
||||||
|
builder.Services.AddSingleton<BlenderRunner>();
|
||||||
builder.Services.AddSingleton<JobExecutor>();
|
builder.Services.AddSingleton<JobExecutor>();
|
||||||
builder.Services.AddHostedService<WorkerService>();
|
builder.Services.AddHostedService<WorkerService>();
|
||||||
|
|
||||||
|
|||||||
@@ -15,7 +15,7 @@
|
|||||||
every change that gets published. publish.ps1 appends the git SHA via
|
every change that gets published. publish.ps1 appends the git SHA via
|
||||||
SourceRevisionId, giving "1.1.0+feb1c7b".
|
SourceRevisionId, giving "1.1.0+feb1c7b".
|
||||||
-->
|
-->
|
||||||
<Version>1.4.0</Version>
|
<Version>1.5.0</Version>
|
||||||
</PropertyGroup>
|
</PropertyGroup>
|
||||||
|
|
||||||
<ItemGroup>
|
<ItemGroup>
|
||||||
|
|||||||
@@ -20,6 +20,12 @@ public sealed class WorkerOptions
|
|||||||
/// <summary>vfxr_build_preview.jsx — defaults to scripts\ beside the worker exe.</summary>
|
/// <summary>vfxr_build_preview.jsx — defaults to scripts\ beside the worker exe.</summary>
|
||||||
public string? PreviewScriptPath { get; set; }
|
public string? PreviewScriptPath { get; set; }
|
||||||
public string? FfmpegPath { get; set; }
|
public string? FfmpegPath { get; set; }
|
||||||
|
/// <summary>
|
||||||
|
/// blender.exe. Its presence is what makes this machine eligible for
|
||||||
|
/// Blender jobs: the version is reported at registration and the server
|
||||||
|
/// only hands BLENDER_RENDER work to machines that have it.
|
||||||
|
/// </summary>
|
||||||
|
public string? BlenderPath { get; set; }
|
||||||
public string? OiiotoolPath { get; set; }
|
public string? OiiotoolPath { get; set; }
|
||||||
public List<PathMapping> PathMappings { get; set; } = new();
|
public List<PathMapping> PathMappings { get; set; } = new();
|
||||||
/// <summary>
|
/// <summary>
|
||||||
|
|||||||
@@ -64,9 +64,11 @@ public sealed class WorkerService : BackgroundService
|
|||||||
}
|
}
|
||||||
// Delivery builds are file work with no AE involvement, so they
|
// Delivery builds are file work with no AE involvement, so they
|
||||||
// are always claimable — someone is usually waiting on the folder.
|
// are always claimable — someone is usually waiting on the folder.
|
||||||
|
// Blender is asked for unconditionally; the server refuses it to
|
||||||
|
// machines whose capabilities do not include Blender.
|
||||||
var types = aeOpen
|
var types = aeOpen
|
||||||
? new[] { "AE_RENDER", "DELIVERY_BUILD" }
|
? new[] { "AE_RENDER", "DELIVERY_BUILD", "BLENDER_RENDER" }
|
||||||
: new[] { "AE_RENDER", "PREVIEW_ONLY", "DELIVERY_BUILD" };
|
: new[] { "AE_RENDER", "PREVIEW_ONLY", "DELIVERY_BUILD", "BLENDER_RENDER" };
|
||||||
|
|
||||||
var job = await _api.ClaimAsync(_machineId!, types, stoppingToken);
|
var job = await _api.ClaimAsync(_machineId!, types, stoppingToken);
|
||||||
if (job is not null)
|
if (job is not null)
|
||||||
|
|||||||
@@ -112,6 +112,12 @@ export function PipelineQueueClient({ projects }: Props) {
|
|||||||
</p>
|
</p>
|
||||||
</div>
|
</div>
|
||||||
<div className="flex items-center gap-2">
|
<div className="flex items-center gap-2">
|
||||||
|
<Link href="/pipeline/deliveries">
|
||||||
|
<Button variant="outline" size="sm">
|
||||||
|
<Package className="h-4 w-4 mr-2" />
|
||||||
|
Deliveries
|
||||||
|
</Button>
|
||||||
|
</Link>
|
||||||
<Link href="/pipeline/deliveries">
|
<Link href="/pipeline/deliveries">
|
||||||
<Button variant="outline" size="sm">
|
<Button variant="outline" size="sm">
|
||||||
<Package className="h-4 w-4 mr-2" />
|
<Package className="h-4 w-4 mr-2" />
|
||||||
@@ -274,6 +280,124 @@ export function PipelineQueueClient({ projects }: Props) {
|
|||||||
</tbody>
|
</tbody>
|
||||||
</table>
|
</table>
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
|
<BlenderQueue projectId={projectId} />
|
||||||
|
</div>
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
interface BlenderJobRow {
|
||||||
|
id: string;
|
||||||
|
label: string | null;
|
||||||
|
status: string;
|
||||||
|
attempt: number;
|
||||||
|
maxAttempts: number;
|
||||||
|
progress: number;
|
||||||
|
currentFrame: number | null;
|
||||||
|
totalFrames: number | null;
|
||||||
|
etaSeconds: number | null;
|
||||||
|
errorMessage: string | null;
|
||||||
|
machineName: string | null;
|
||||||
|
submittedBy: string | null;
|
||||||
|
projectCode: string | null;
|
||||||
|
blendFile: string | null;
|
||||||
|
frameStart: number | null;
|
||||||
|
frameEnd: number | null;
|
||||||
|
createdAt: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
const BLENDER_STATUS_STYLES: Record<string, string> = {
|
||||||
|
QUEUED: "bg-zinc-500/15 text-zinc-300 border-zinc-500/30",
|
||||||
|
CLAIMED: "bg-blue-500/15 text-blue-400 border-blue-500/30",
|
||||||
|
RUNNING: "bg-blue-500/15 text-blue-400 border-blue-500/30",
|
||||||
|
COMPLETED: "bg-emerald-500/15 text-emerald-400 border-emerald-500/30",
|
||||||
|
FAILED: "bg-red-500/15 text-red-400 border-red-500/30",
|
||||||
|
CANCELLED: "bg-zinc-500/10 text-zinc-500 border-zinc-600/30",
|
||||||
|
EXPIRED: "bg-red-500/15 text-red-400 border-red-500/30",
|
||||||
|
};
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Standalone Blender renders. They have no Export behind them, so they cannot
|
||||||
|
* appear in the table above — but they share the queue, and a render nobody can
|
||||||
|
* see is a render nobody can retry.
|
||||||
|
*/
|
||||||
|
function BlenderQueue({ projectId }: { projectId: string }) {
|
||||||
|
const { toast } = useToast();
|
||||||
|
const queryClient = useQueryClient();
|
||||||
|
|
||||||
|
const { data } = useQuery({
|
||||||
|
queryKey: ["blender-jobs", projectId],
|
||||||
|
queryFn: async () => {
|
||||||
|
const qs = projectId && projectId !== "all" ? `?projectId=${encodeURIComponent(projectId)}` : "";
|
||||||
|
const res = await fetch(`/api/render/blender${qs}`);
|
||||||
|
if (!res.ok) throw new Error("Failed to load Blender jobs");
|
||||||
|
return res.json() as Promise<{ jobs: BlenderJobRow[] }>;
|
||||||
|
},
|
||||||
|
refetchInterval: 5000,
|
||||||
|
});
|
||||||
|
|
||||||
|
async function jobAction(id: string, verb: "cancel" | "retry") {
|
||||||
|
const res = await fetch(`/api/render/blender/${id}?action=${verb}`, { method: "POST" });
|
||||||
|
if (!res.ok) {
|
||||||
|
const body = await res.json().catch(() => ({}));
|
||||||
|
toast({ title: `${verb} failed`, description: body.error ?? res.statusText, variant: "destructive" });
|
||||||
|
}
|
||||||
|
queryClient.invalidateQueries({ queryKey: ["blender-jobs"] });
|
||||||
|
}
|
||||||
|
|
||||||
|
const jobs = data?.jobs ?? [];
|
||||||
|
if (jobs.length === 0) return null;
|
||||||
|
|
||||||
|
return (
|
||||||
|
<div className="rounded-lg border border-zinc-800 bg-zinc-900/40">
|
||||||
|
<div className="px-4 py-2.5 border-b border-zinc-800 text-sm font-medium text-white">
|
||||||
|
Blender renders
|
||||||
|
</div>
|
||||||
|
<div className="divide-y divide-zinc-800/60">
|
||||||
|
{jobs.map((j) => (
|
||||||
|
<div key={j.id} className="px-4 py-3 text-sm">
|
||||||
|
<div className="flex items-center gap-3 flex-wrap">
|
||||||
|
<span className="text-zinc-200">{j.label ?? "(untitled)"}</span>
|
||||||
|
<span className={cn("rounded-full border px-2 py-0.5 text-xs", BLENDER_STATUS_STYLES[j.status] ?? "")}>
|
||||||
|
{j.status.toLowerCase()}
|
||||||
|
</span>
|
||||||
|
{j.projectCode && <span className="text-xs text-zinc-500">{j.projectCode}</span>}
|
||||||
|
{j.frameStart != null && (
|
||||||
|
<span className="text-xs text-zinc-500">
|
||||||
|
frames {j.frameStart}–{j.frameEnd}
|
||||||
|
</span>
|
||||||
|
)}
|
||||||
|
{j.status === "RUNNING" && (
|
||||||
|
<span className="text-xs text-blue-400">
|
||||||
|
{Math.round(j.progress * 100)}%
|
||||||
|
{j.currentFrame != null ? ` · frame ${j.currentFrame}` : ""}
|
||||||
|
{j.etaSeconds ? ` · ETA ${formatEta(j.etaSeconds)}` : ""}
|
||||||
|
</span>
|
||||||
|
)}
|
||||||
|
{j.machineName && <span className="text-xs text-zinc-500">{j.machineName}</span>}
|
||||||
|
{j.attempt > 1 && (
|
||||||
|
<span className="text-xs text-zinc-500">
|
||||||
|
attempt {j.attempt}/{j.maxAttempts}
|
||||||
|
</span>
|
||||||
|
)}
|
||||||
|
<span className="ml-auto flex items-center gap-1">
|
||||||
|
{["QUEUED", "CLAIMED", "RUNNING"].includes(j.status) && (
|
||||||
|
<Button variant="ghost" size="icon-sm" title="Cancel" onClick={() => jobAction(j.id, "cancel")}>
|
||||||
|
<XCircle className="h-4 w-4 text-red-400" />
|
||||||
|
</Button>
|
||||||
|
)}
|
||||||
|
{["FAILED", "CANCELLED", "EXPIRED"].includes(j.status) && (
|
||||||
|
<Button variant="ghost" size="icon-sm" title="Requeue" onClick={() => jobAction(j.id, "retry")}>
|
||||||
|
<RotateCcw className="h-4 w-4 text-amber-400" />
|
||||||
|
</Button>
|
||||||
|
)}
|
||||||
|
</span>
|
||||||
|
</div>
|
||||||
|
{j.blendFile && <div className="mt-1 text-xs text-zinc-600 break-all">{j.blendFile}</div>}
|
||||||
|
{j.errorMessage && <div className="mt-1 text-xs text-red-400">{j.errorMessage}</div>}
|
||||||
|
</div>
|
||||||
|
))}
|
||||||
|
</div>
|
||||||
</div>
|
</div>
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -0,0 +1,36 @@
|
|||||||
|
import { NextRequest, NextResponse } from "next/server";
|
||||||
|
import { isExtAuthorized } from "@/lib/render-pipeline/ext-auth";
|
||||||
|
import { toErrorResponse } from "@/lib/render-pipeline/errors";
|
||||||
|
import { createBlenderJob, createBlenderJobSchema } from "@/lib/render-pipeline/blender";
|
||||||
|
|
||||||
|
// ── POST /api/ext/render/blender ─────────────────────────────────────────────
|
||||||
|
//
|
||||||
|
// Queue a standalone Blender render from the Blender add-on. No Export, no shot
|
||||||
|
// version, no validation or QC — the .blend is already set up to render, and
|
||||||
|
// this puts it on the farm instead of on the artist's machine.
|
||||||
|
|
||||||
|
export async function POST(req: NextRequest) {
|
||||||
|
if (!isExtAuthorized(req)) {
|
||||||
|
return NextResponse.json({ error: "Unauthorized" }, { status: 401 });
|
||||||
|
}
|
||||||
|
|
||||||
|
let raw: unknown;
|
||||||
|
try {
|
||||||
|
raw = await req.json();
|
||||||
|
} catch {
|
||||||
|
return NextResponse.json({ error: "Invalid JSON" }, { status: 400 });
|
||||||
|
}
|
||||||
|
|
||||||
|
const parsed = createBlenderJobSchema.safeParse(raw);
|
||||||
|
if (!parsed.success) {
|
||||||
|
return NextResponse.json({ error: "Invalid Blender job", details: parsed.error.issues }, { status: 422 });
|
||||||
|
}
|
||||||
|
|
||||||
|
try {
|
||||||
|
const result = await createBlenderJob(parsed.data);
|
||||||
|
return NextResponse.json(result, { status: 201 });
|
||||||
|
} catch (err) {
|
||||||
|
const { body, status } = toErrorResponse(err);
|
||||||
|
return NextResponse.json(body, { status });
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,28 @@
|
|||||||
|
import { NextRequest, NextResponse } from "next/server";
|
||||||
|
import { toErrorResponse } from "@/lib/render-pipeline/errors";
|
||||||
|
import { requirePipelineUser } from "@/lib/render-pipeline/session-auth";
|
||||||
|
import { cancelBlenderJob, retryBlenderJob } from "@/lib/render-pipeline/blender";
|
||||||
|
|
||||||
|
// ── POST /api/render/blender/{jobId}?action=cancel|retry ─────────────────────
|
||||||
|
|
||||||
|
export async function POST(
|
||||||
|
req: NextRequest,
|
||||||
|
{ params }: { params: Promise<{ jobId: string }> }
|
||||||
|
) {
|
||||||
|
try {
|
||||||
|
const user = await requirePipelineUser();
|
||||||
|
const { jobId } = await params;
|
||||||
|
const action = new URL(req.url).searchParams.get("action");
|
||||||
|
|
||||||
|
if (action === "retry") {
|
||||||
|
return NextResponse.json(await retryBlenderJob(jobId));
|
||||||
|
}
|
||||||
|
if (action === "cancel") {
|
||||||
|
return NextResponse.json(await cancelBlenderJob(jobId, user.id));
|
||||||
|
}
|
||||||
|
return NextResponse.json({ error: "action must be cancel or retry" }, { status: 400 });
|
||||||
|
} catch (err) {
|
||||||
|
const { body, status } = toErrorResponse(err);
|
||||||
|
return NextResponse.json(body, { status });
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,24 @@
|
|||||||
|
import { NextRequest, NextResponse } from "next/server";
|
||||||
|
import { RenderJobStatus } from "@prisma/client";
|
||||||
|
import { toErrorResponse } from "@/lib/render-pipeline/errors";
|
||||||
|
import { requirePipelineUser } from "@/lib/render-pipeline/session-auth";
|
||||||
|
import { listBlenderJobs } from "@/lib/render-pipeline/blender";
|
||||||
|
|
||||||
|
// ── GET /api/render/blender?projectId=&status= ───────────────────────────────
|
||||||
|
|
||||||
|
export async function GET(req: NextRequest) {
|
||||||
|
try {
|
||||||
|
await requirePipelineUser();
|
||||||
|
const { searchParams } = new URL(req.url);
|
||||||
|
const status = searchParams.get("status");
|
||||||
|
const result = await listBlenderJobs({
|
||||||
|
projectId: searchParams.get("projectId") ?? undefined,
|
||||||
|
status: status ? (status as RenderJobStatus) : undefined,
|
||||||
|
limit: searchParams.get("limit") ? Number(searchParams.get("limit")) : undefined,
|
||||||
|
});
|
||||||
|
return NextResponse.json(result);
|
||||||
|
} catch (err) {
|
||||||
|
const { body, status } = toErrorResponse(err);
|
||||||
|
return NextResponse.json(body, { status });
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,50 @@
|
|||||||
|
import { describe, it, expect } from "vitest";
|
||||||
|
import { blenderManifestSchema, createBlenderJobSchema } from "@/lib/render-pipeline/blender";
|
||||||
|
|
||||||
|
/**
|
||||||
|
* The add-on and the server are separate codebases that only meet over JSON,
|
||||||
|
* so the exact payload the add-on builds is pinned here. A field renamed on
|
||||||
|
* either side otherwise surfaces as a 422 in front of an artist.
|
||||||
|
*/
|
||||||
|
describe("Blender submission contract", () => {
|
||||||
|
// Verbatim shape of scene_summary() in BlenderPanel/vfxreview_submit.py
|
||||||
|
const addonManifest = {
|
||||||
|
blendFile: "X:/shows/UNG/cg/UNG_110_030_010_smoke.blend",
|
||||||
|
scene: "Scene",
|
||||||
|
frameStart: 1001,
|
||||||
|
frameEnd: 1110,
|
||||||
|
outputPath: "X:/shows/UNG/cg/renders/smoke/frame_",
|
||||||
|
engine: "CYCLES",
|
||||||
|
blenderVersion: "4.2.1",
|
||||||
|
};
|
||||||
|
|
||||||
|
it("accepts what the add-on sends", () => {
|
||||||
|
const parsed = blenderManifestSchema.safeParse(addonManifest);
|
||||||
|
expect(parsed.success).toBe(true);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("accepts the full submission body", () => {
|
||||||
|
const parsed = createBlenderJobSchema.safeParse({
|
||||||
|
projectCode: "UNG_S1",
|
||||||
|
label: "UNG_110_030_010_smoke",
|
||||||
|
manifest: addonManifest,
|
||||||
|
submittedByEmail: "chris@twotalesanimation.com",
|
||||||
|
urgent: false,
|
||||||
|
});
|
||||||
|
expect(parsed.success).toBe(true);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("defaults the stage so the add-on need not send it", () => {
|
||||||
|
const parsed = blenderManifestSchema.parse(addonManifest);
|
||||||
|
expect(parsed.stage).toBe("BLENDER");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("rejects a backwards frame range rather than queueing a no-op render", () => {
|
||||||
|
const parsed = blenderManifestSchema.safeParse({ ...addonManifest, frameEnd: 900 });
|
||||||
|
expect(parsed.success).toBe(false);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("accepts EEVEE, which is what the add-on reports on a non-Cycles scene", () => {
|
||||||
|
expect(blenderManifestSchema.safeParse({ ...addonManifest, engine: "BLENDER_EEVEE_NEXT" }).success).toBe(true);
|
||||||
|
});
|
||||||
|
});
|
||||||
@@ -0,0 +1,189 @@
|
|||||||
|
import { Prisma, RenderJobStatus } from "@prisma/client";
|
||||||
|
import { z } from "zod";
|
||||||
|
import { db } from "@/lib/db";
|
||||||
|
import { PipelineError } from "./errors";
|
||||||
|
import { getConfigNumber } from "./config";
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Standalone Blender renders.
|
||||||
|
*
|
||||||
|
* Deliberately the simplest thing in the pipeline: a queue entry and nothing
|
||||||
|
* more. No Export, no shot version, no validation, QC or delivery — the .blend
|
||||||
|
* is already set up to render, and this is "Ctrl+F12, but on the farm". Every
|
||||||
|
* lifecycle call it needs (claim, progress, fail, complete) already tolerates a
|
||||||
|
* job with no Export behind it.
|
||||||
|
*/
|
||||||
|
|
||||||
|
export const blenderManifestSchema = z.object({
|
||||||
|
stage: z.literal("BLENDER").default("BLENDER"),
|
||||||
|
blendFile: z.string().min(1),
|
||||||
|
/** Blank renders the .blend's active scene, which is usually what is wanted. */
|
||||||
|
scene: z.string().max(200).optional().nullable(),
|
||||||
|
frameStart: z.number().int(),
|
||||||
|
frameEnd: z.number().int(),
|
||||||
|
/**
|
||||||
|
* Read from the .blend at submission and passed through only so the queue can
|
||||||
|
* show where frames will land. Left unset, the file's own output path is used
|
||||||
|
* — the file is expected to be ready to render.
|
||||||
|
*/
|
||||||
|
outputPath: z.string().max(1000).optional().nullable(),
|
||||||
|
/** Overrides, all optional: an unset value leaves the .blend's own setting. */
|
||||||
|
engine: z.enum(["CYCLES", "BLENDER_EEVEE_NEXT", "BLENDER_EEVEE", "BLENDER_WORKBENCH"]).optional().nullable(),
|
||||||
|
device: z.enum(["CPU", "GPU"]).optional().nullable(),
|
||||||
|
samples: z.number().int().positive().max(100000).optional().nullable(),
|
||||||
|
resolutionPercent: z.number().int().min(1).max(100).optional().nullable(),
|
||||||
|
/** Blender version this was authored in, for diagnosing a farm-vs-desk difference. */
|
||||||
|
blenderVersion: z.string().max(50).optional().nullable(),
|
||||||
|
}).refine((m) => m.frameEnd >= m.frameStart, {
|
||||||
|
message: "frameEnd must be >= frameStart",
|
||||||
|
path: ["frameEnd"],
|
||||||
|
});
|
||||||
|
|
||||||
|
export type BlenderManifest = z.infer<typeof blenderManifestSchema>;
|
||||||
|
|
||||||
|
export const createBlenderJobSchema = z.object({
|
||||||
|
projectCode: z.string().min(1),
|
||||||
|
label: z.string().min(1).max(200),
|
||||||
|
manifest: blenderManifestSchema,
|
||||||
|
submittedByEmail: z.string().max(200).optional().nullable(),
|
||||||
|
urgent: z.boolean().optional(),
|
||||||
|
});
|
||||||
|
|
||||||
|
export type CreateBlenderJobInput = z.infer<typeof createBlenderJobSchema>;
|
||||||
|
|
||||||
|
export async function createBlenderJob(input: CreateBlenderJobInput) {
|
||||||
|
const project = await db.project.findUnique({
|
||||||
|
where: { code: input.projectCode },
|
||||||
|
select: { id: true, code: true, name: true },
|
||||||
|
});
|
||||||
|
if (!project) throw new PipelineError(404, `Project not found: ${input.projectCode}`);
|
||||||
|
|
||||||
|
const [maxAttempts, urgentThreshold] = await Promise.all([
|
||||||
|
getConfigNumber("render.maxAttempts"),
|
||||||
|
getConfigNumber("render.urgentPriorityThreshold"),
|
||||||
|
]);
|
||||||
|
|
||||||
|
const job = await db.renderJob.create({
|
||||||
|
data: {
|
||||||
|
type: "BLENDER_RENDER",
|
||||||
|
projectId: project.id,
|
||||||
|
label: input.label.trim(),
|
||||||
|
submittedBy: input.submittedByEmail?.trim() || null,
|
||||||
|
attempt: 1,
|
||||||
|
maxAttempts,
|
||||||
|
priority: input.urgent ? urgentThreshold : 50,
|
||||||
|
manifest: input.manifest as unknown as Prisma.InputJsonValue,
|
||||||
|
},
|
||||||
|
});
|
||||||
|
|
||||||
|
return {
|
||||||
|
job: {
|
||||||
|
id: job.id,
|
||||||
|
label: job.label,
|
||||||
|
status: job.status,
|
||||||
|
priority: job.priority,
|
||||||
|
frames: input.manifest.frameEnd - input.manifest.frameStart + 1,
|
||||||
|
},
|
||||||
|
project: { code: project.code, name: project.name },
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface BlenderJobFilter {
|
||||||
|
projectId?: string;
|
||||||
|
status?: RenderJobStatus;
|
||||||
|
limit?: number;
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function listBlenderJobs(filter: BlenderJobFilter = {}) {
|
||||||
|
const jobs = await db.renderJob.findMany({
|
||||||
|
where: {
|
||||||
|
type: "BLENDER_RENDER",
|
||||||
|
...(filter.projectId ? { projectId: filter.projectId } : {}),
|
||||||
|
...(filter.status ? { status: filter.status } : {}),
|
||||||
|
},
|
||||||
|
orderBy: [{ createdAt: "desc" }],
|
||||||
|
take: Math.min(200, Math.max(1, filter.limit ?? 50)),
|
||||||
|
include: { machine: { select: { name: true } } },
|
||||||
|
});
|
||||||
|
|
||||||
|
const projectIds = [...new Set(jobs.map((j) => j.projectId).filter((id): id is string => !!id))];
|
||||||
|
const projects = projectIds.length
|
||||||
|
? await db.project.findMany({ where: { id: { in: projectIds } }, select: { id: true, code: true, name: true } })
|
||||||
|
: [];
|
||||||
|
const byId = new Map(projects.map((p) => [p.id, p]));
|
||||||
|
|
||||||
|
return {
|
||||||
|
jobs: jobs.map((j) => {
|
||||||
|
const m = j.manifest as unknown as Partial<BlenderManifest>;
|
||||||
|
return {
|
||||||
|
id: j.id,
|
||||||
|
label: j.label,
|
||||||
|
status: j.status,
|
||||||
|
attempt: j.attempt,
|
||||||
|
maxAttempts: j.maxAttempts,
|
||||||
|
priority: j.priority,
|
||||||
|
progress: j.progress,
|
||||||
|
currentFrame: j.currentFrame,
|
||||||
|
totalFrames: j.totalFrames,
|
||||||
|
etaSeconds: j.etaSeconds,
|
||||||
|
errorMessage: j.errorMessage,
|
||||||
|
machineName: j.machine?.name ?? null,
|
||||||
|
submittedBy: j.submittedBy,
|
||||||
|
createdAt: j.createdAt,
|
||||||
|
startedAt: j.startedAt,
|
||||||
|
finishedAt: j.finishedAt,
|
||||||
|
projectCode: j.projectId ? byId.get(j.projectId)?.code ?? null : null,
|
||||||
|
blendFile: m.blendFile ?? null,
|
||||||
|
outputPath: m.outputPath ?? null,
|
||||||
|
frameStart: m.frameStart ?? null,
|
||||||
|
frameEnd: m.frameEnd ?? null,
|
||||||
|
};
|
||||||
|
}),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Cancel a queued or running Blender job. The worker learns via its progress report. */
|
||||||
|
export async function cancelBlenderJob(jobId: string, actorId?: string) {
|
||||||
|
const job = await db.renderJob.findUnique({ where: { id: jobId }, select: { id: true, type: true, status: true } });
|
||||||
|
if (!job) throw new PipelineError(404, "Render job not found");
|
||||||
|
if (job.type !== "BLENDER_RENDER") throw new PipelineError(422, "Not a Blender job");
|
||||||
|
if (!["QUEUED", "CLAIMED", "RUNNING"].includes(job.status)) {
|
||||||
|
throw new PipelineError(409, `Job is ${job.status} — only a queued or running job can be cancelled`);
|
||||||
|
}
|
||||||
|
|
||||||
|
const updated = await db.renderJob.update({
|
||||||
|
where: { id: job.id },
|
||||||
|
data: {
|
||||||
|
status: "CANCELLED",
|
||||||
|
errorMessage: `Cancelled by user${actorId ? ` ${actorId}` : ""}`,
|
||||||
|
// A claimed job keeps finishedAt null until the worker acknowledges, so a
|
||||||
|
// render still winding down is not reported as already stopped.
|
||||||
|
...(job.status === "QUEUED" ? { finishedAt: new Date() } : {}),
|
||||||
|
},
|
||||||
|
});
|
||||||
|
return { job: { id: updated.id, status: updated.status } };
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Requeue a failed Blender job as a fresh attempt, reusing its manifest. */
|
||||||
|
export async function retryBlenderJob(jobId: string) {
|
||||||
|
const job = await db.renderJob.findUnique({ where: { id: jobId } });
|
||||||
|
if (!job) throw new PipelineError(404, "Render job not found");
|
||||||
|
if (job.type !== "BLENDER_RENDER") throw new PipelineError(422, "Not a Blender job");
|
||||||
|
if (!["FAILED", "CANCELLED", "EXPIRED"].includes(job.status)) {
|
||||||
|
throw new PipelineError(409, `Job is ${job.status} — only a finished-unsuccessfully job can be retried`);
|
||||||
|
}
|
||||||
|
|
||||||
|
const next = await db.renderJob.create({
|
||||||
|
data: {
|
||||||
|
type: "BLENDER_RENDER",
|
||||||
|
projectId: job.projectId,
|
||||||
|
label: job.label,
|
||||||
|
submittedBy: job.submittedBy,
|
||||||
|
attempt: job.attempt + 1,
|
||||||
|
maxAttempts: job.maxAttempts,
|
||||||
|
priority: job.priority,
|
||||||
|
manifest: job.manifest as Prisma.InputJsonValue,
|
||||||
|
},
|
||||||
|
});
|
||||||
|
return { job: { id: next.id, attempt: next.attempt, status: next.status } };
|
||||||
|
}
|
||||||
@@ -3,11 +3,11 @@ import { db } from "@/lib/db";
|
|||||||
import { PipelineError } from "./errors";
|
import { PipelineError } from "./errors";
|
||||||
import { applyTransition } from "./transitions";
|
import { applyTransition } from "./transitions";
|
||||||
import { getConfigBoolean, getConfigNumber } from "./config";
|
import { getConfigBoolean, getConfigNumber } from "./config";
|
||||||
import { machineCanClaim } from "./machines";
|
import { machineCanClaim, machineSupportedTypes } from "./machines";
|
||||||
import { reapExpiredLeases } from "./reaper";
|
import { reapExpiredLeases } from "./reaper";
|
||||||
import { buildPreviewManifest, createPreviewJob, inheritPreviewMedia } from "./preview";
|
import { buildPreviewManifest, createPreviewJob, inheritPreviewMedia } from "./preview";
|
||||||
|
|
||||||
const JOB_TYPES: RenderJobType[] = ["AE_RENDER", "PREVIEW_ONLY", "DELIVERY_BUILD"];
|
const JOB_TYPES: RenderJobType[] = ["AE_RENDER", "PREVIEW_ONLY", "DELIVERY_BUILD", "BLENDER_RENDER"];
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* E8 — atomically claim the next queued job (§6.4). `FOR UPDATE SKIP LOCKED`
|
* E8 — atomically claim the next queued job (§6.4). `FOR UPDATE SKIP LOCKED`
|
||||||
@@ -22,10 +22,14 @@ export async function claimNextJob(machineId: string, types?: string[]) {
|
|||||||
// Defensive sweep on each claim (studio decision 18.1-Q8)
|
// Defensive sweep on each claim (studio decision 18.1-Q8)
|
||||||
await reapExpiredLeases().catch((err) => console.error("[render-pipeline] claim-time reap failed", err));
|
await reapExpiredLeases().catch((err) => console.error("[render-pipeline] claim-time reap failed", err));
|
||||||
|
|
||||||
const claimTypes = (types?.length ? types : ["AE_RENDER", "PREVIEW_ONLY", "DELIVERY_BUILD"]).filter(
|
const requested = (types?.length ? types : ["AE_RENDER", "PREVIEW_ONLY", "DELIVERY_BUILD"]).filter(
|
||||||
(t): t is RenderJobType => (JOB_TYPES as string[]).includes(t)
|
(t): t is RenderJobType => (JOB_TYPES as string[]).includes(t)
|
||||||
);
|
);
|
||||||
if (claimTypes.length === 0) throw new PipelineError(422, "No valid job types requested");
|
// Enforced here rather than trusted to the worker: a machine without the tool
|
||||||
|
// installed must not be able to claim work it can only fail.
|
||||||
|
const claimTypes = machineSupportedTypes(machine, requested) as RenderJobType[];
|
||||||
|
if (requested.length === 0) throw new PipelineError(422, "No valid job types requested");
|
||||||
|
if (claimTypes.length === 0) return null;
|
||||||
|
|
||||||
const gate = machineCanClaim(machine, new Date());
|
const gate = machineCanClaim(machine, new Date());
|
||||||
if (gate === "none") return null;
|
if (gate === "none") return null;
|
||||||
@@ -91,6 +95,7 @@ export async function claimNextJob(machineId: string, types?: string[]) {
|
|||||||
priority: job.priority,
|
priority: job.priority,
|
||||||
exportId: job.exportId,
|
exportId: job.exportId,
|
||||||
deliveryId: job.deliveryId,
|
deliveryId: job.deliveryId,
|
||||||
|
label: job.label,
|
||||||
leaseExpiresAt: job.leaseExpiresAt,
|
leaseExpiresAt: job.leaseExpiresAt,
|
||||||
manifest: job.manifest,
|
manifest: job.manifest,
|
||||||
},
|
},
|
||||||
@@ -189,6 +194,33 @@ export async function reportFail(
|
|||||||
});
|
});
|
||||||
|
|
||||||
if (!job.export) {
|
if (!job.export) {
|
||||||
|
// A standalone job (a Blender render) has no Export to move, but a
|
||||||
|
// transient failure should still be retried like any other work.
|
||||||
|
// Delivery builds are deliberately excluded: a half-written package
|
||||||
|
// folder wants a human to look at it before it is rebuilt.
|
||||||
|
const retryableStandalone =
|
||||||
|
!job.deliveryId && body.retryable !== false && job.attempt < job.maxAttempts;
|
||||||
|
if (retryableStandalone) {
|
||||||
|
const next = await tx.renderJob.create({
|
||||||
|
data: {
|
||||||
|
type: job.type,
|
||||||
|
projectId: job.projectId,
|
||||||
|
label: job.label,
|
||||||
|
submittedBy: job.submittedBy,
|
||||||
|
deliveryId: job.deliveryId,
|
||||||
|
attempt: job.attempt + 1,
|
||||||
|
maxAttempts: job.maxAttempts,
|
||||||
|
priority: job.priority,
|
||||||
|
manifest: job.manifest as Prisma.InputJsonValue,
|
||||||
|
},
|
||||||
|
});
|
||||||
|
return {
|
||||||
|
job: { id: job.id, status: "FAILED" },
|
||||||
|
export: null,
|
||||||
|
autoRequeued: true,
|
||||||
|
nextJob: { id: next.id, attempt: next.attempt },
|
||||||
|
};
|
||||||
|
}
|
||||||
return { job: { id: job.id, status: "FAILED" }, export: null, autoRequeued: false };
|
return { job: { id: job.id, status: "FAILED" }, export: null, autoRequeued: false };
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -59,6 +59,19 @@ export function isInWindow(windows: AvailabilityWindow[], now: Date): boolean {
|
|||||||
|
|
||||||
export type ClaimGate = "normal" | "urgent-only" | "none";
|
export type ClaimGate = "normal" | "urgent-only" | "none";
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Job types a machine is equipped for. Capabilities are reported at
|
||||||
|
* registration; a machine without Blender installed must never be handed a
|
||||||
|
* Blender job, or it claims work it can only fail — repeatedly, until the
|
||||||
|
* attempts run out — while a machine that could have rendered it sits idle.
|
||||||
|
*/
|
||||||
|
export function machineSupportedTypes(machine: { capabilities: unknown }, requested: string[]): string[] {
|
||||||
|
const tools = ((machine.capabilities as { tools?: Record<string, unknown> } | null)?.tools ?? {}) as Record<string, unknown>;
|
||||||
|
const hasBlender = Boolean(tools.blender);
|
||||||
|
|
||||||
|
return requested.filter((type) => (type === "BLENDER_RENDER" ? hasBlender : true));
|
||||||
|
}
|
||||||
|
|
||||||
export function machineCanClaim(machine: Machine, now: Date): ClaimGate {
|
export function machineCanClaim(machine: Machine, now: Date): ClaimGate {
|
||||||
if (!machine.enabled) return "none";
|
if (!machine.enabled) return "none";
|
||||||
if (machine.renderNowUntil && machine.renderNowUntil > now) return "normal";
|
if (machine.renderNowUntil && machine.renderNowUntil > now) return "normal";
|
||||||
|
|||||||
@@ -0,0 +1,12 @@
|
|||||||
|
-- Standalone Blender renders: a queue entry with no Export behind it.
|
||||||
|
|
||||||
|
-- AlterEnum
|
||||||
|
ALTER TYPE "RenderJobType" ADD VALUE 'BLENDER_RENDER';
|
||||||
|
|
||||||
|
-- AlterTable
|
||||||
|
ALTER TABLE "render_jobs" ADD COLUMN "projectId" TEXT;
|
||||||
|
ALTER TABLE "render_jobs" ADD COLUMN "label" TEXT;
|
||||||
|
ALTER TABLE "render_jobs" ADD COLUMN "submittedBy" TEXT;
|
||||||
|
|
||||||
|
-- CreateIndex
|
||||||
|
CREATE INDEX "render_jobs_projectId_status_idx" ON "render_jobs"("projectId", "status");
|
||||||
@@ -149,6 +149,10 @@ enum RenderJobType {
|
|||||||
AE_RENDER
|
AE_RENDER
|
||||||
PREVIEW_ONLY
|
PREVIEW_ONLY
|
||||||
DELIVERY_BUILD
|
DELIVERY_BUILD
|
||||||
|
/// A standalone Blender render: a queue entry and nothing more. It has no
|
||||||
|
/// Export, takes no shot version, and is not validated, QC'd or delivered —
|
||||||
|
/// it is "Ctrl+F12, but on the farm".
|
||||||
|
BLENDER_RENDER
|
||||||
}
|
}
|
||||||
|
|
||||||
enum ValidationStatus {
|
enum ValidationStatus {
|
||||||
@@ -915,6 +919,13 @@ model RenderJob {
|
|||||||
exportId String? // set for AE_RENDER / PREVIEW_ONLY
|
exportId String? // set for AE_RENDER / PREVIEW_ONLY
|
||||||
export Export? @relation(fields: [exportId], references: [id])
|
export Export? @relation(fields: [exportId], references: [id])
|
||||||
deliveryId String? // set for DELIVERY_BUILD (Phase 5)
|
deliveryId String? // set for DELIVERY_BUILD (Phase 5)
|
||||||
|
/// Set for jobs that belong to a project but not to a shot — a standalone
|
||||||
|
/// Blender render. Lets the queue be filtered by production without
|
||||||
|
/// inventing an Export for work that has no shot.
|
||||||
|
projectId String?
|
||||||
|
/// Human label for such a job, since there is no shot code to show.
|
||||||
|
label String?
|
||||||
|
submittedBy String? // email/name of whoever queued it
|
||||||
attempt Int @default(1)
|
attempt Int @default(1)
|
||||||
maxAttempts Int @default(3)
|
maxAttempts Int @default(3)
|
||||||
status RenderJobStatus @default(QUEUED)
|
status RenderJobStatus @default(QUEUED)
|
||||||
@@ -942,6 +953,7 @@ model RenderJob {
|
|||||||
|
|
||||||
@@index([status, priority, createdAt])
|
@@index([status, priority, createdAt])
|
||||||
@@index([machineId, status])
|
@@index([machineId, status])
|
||||||
|
@@index([projectId, status])
|
||||||
@@map("render_jobs")
|
@@map("render_jobs")
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user