feat(render-pipeline): validation, QC and delivery builder
Completes RenderPipeline2 phases 3-5. Additive throughout: with validation.enabled=false the pipeline behaves exactly as before. Validation (§8) runs on the worker straight after a render, while it still holds the job and the files are local. Everything is read from the EXR header — one 64 KB read per sampled frame, no pixel decode — so the engine needs no oiiotool. The render job deliberately stays open through the stage, so a worker that dies there is reaped and retried instead of stranding the export in VALIDATING. Timecode is the primary check and is anchored on the plate, never the API: measured on the live show, Shot.timecodeStart is the editorial in-point and sits one handle-length (8f) ahead of the plate, so checking against it would pass the defective exports and fail the correct ones. Reference precedence is manifest timing plate -> plate found on disk -> API value, and the API rank is WARN-only. The panel now resolves plate provenance by walking the comp's own footage layers, so the worker never guesses which element carries the comp's timing. Verified against real studio renders via the new --validate mode: it catches both defect classes seen in shipped work — the +8 handle offset and frame-derived timecode (00:00:41:17 = frame 1001 at 24fps). QC (§10) is one service behind two front doors, API-key for the panel and session for the web. PASS means ready to upload and nothing else; FAIL sets the comp task to CHANGES. The panel gains a QC section that imports the real EXR sequence at full res rather than the H.264. Delivery (§11) rides the existing queue as a DELIVERY_BUILD job: hard links with a copy fallback, verification by re-hashing delivered bytes against the validation-time digest, manifest.json and the CSV report. fsutil hardlink list is unavailable over SMB, so link identity is never queried — reading the bytes back covers links and copies alike. Also fixes ApiClient.ProgressAsync throwing on transport failure, which would have aborted work in flight during a brief server outage; progress reporting is best-effort by contract. Removes the stale RenderWorker/ rule from the root .gitignore. The worker's source is tracked like the rest of the repo and its own .gitignore already covers bin/, obj/ and publish/; the rule silently excluded new files while the existing 20 stayed tracked. Not yet applied: migration 20260813000000 was hand-written because the dev database was unavailable — run prisma migrate deploy and verify before relying on it. None of this has run against a live server. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
+44
-2
@@ -60,8 +60,47 @@ the artist's AEP (then it is pure `aerender`), or the ffmpeg engine.
|
||||
are never resumed — the next attempt clears its own output files and re-renders.
|
||||
- Durable reporting: complete/fail reports spool to disk and replay in order with backoff —
|
||||
rendering continues while the server is down.
|
||||
- **Validation (Phase 3)**: after a successful render the worker checks the sequence it just
|
||||
produced and reports every result in one call (E12). See below.
|
||||
|
||||
Preview generation (Phase 4) and validation (Phase 3) plug into this same service later.
|
||||
## Validation stage
|
||||
|
||||
Runs on the worker straight after the render, while it still holds the job and the files are
|
||||
local. The Export sits in `VALIDATING` until the report lands; the server decides what the
|
||||
results mean (any FAIL → `VALIDATION_FAILED`, otherwise the preview stage is queued).
|
||||
|
||||
Checks: `file_count`, `missing_frames`, `duplicate_frames`, `first_frame`, `last_frame`,
|
||||
`filename_consistency`, `zero_byte`, `corrupt`, `resolution`, `bit_depth`, `alpha_channel`,
|
||||
`exr_type`, `timecode_plate_match`, `timecode_consistency`, `timecode_fps`.
|
||||
|
||||
**No oiiotool needed.** Everything is read from the EXR header — one 64 KB read per sampled
|
||||
frame, no pixel decode. Full-sequence checks (count, gaps, names, sizes) touch every file;
|
||||
header checks sample first/middle/last plus every Nth frame (`validation.sampleEvery`).
|
||||
|
||||
**Timecode is the primary check** (spec §8.1a) and is compared against *the plate*, in this
|
||||
order: the timing plate named in the manifest → a plate found under `plateSearchRoots` →
|
||||
the API timecode minus handles, which is **WARN-only** because the API stores the editorial
|
||||
in-point and is a handle-length ahead of the plate by design.
|
||||
|
||||
Checksums are xxHash64 per file, with a sequence digest stored on the Export and reused by
|
||||
the delivery builder to verify what it packaged.
|
||||
|
||||
Server-side switches (SystemConfig, no worker redeploy): `validation.enabled`,
|
||||
`validation.sampleEvery`, `validation.handleFrames`, and `validation.overrides` for per-check
|
||||
severity, e.g. `{"exr_type":"warn"}`.
|
||||
|
||||
### Checking a render by hand
|
||||
|
||||
```powershell
|
||||
.\VFXReviewWorker.exe --validate "V:\_EXPORTS\UNG\102\UNG_102_019_010\v005" --plate "V:\_FOOTAGE\UNG\UNG_102\UNG_102_019_010_BG01_TT_v001\UNG_102_019_010_BG01_TT_v001.01001.exr"
|
||||
```
|
||||
|
||||
Runs the whole engine over an existing folder with no server, job or queue, and prints the
|
||||
report. The frame range, pattern and resolution are inferred from the files. Exits non-zero
|
||||
if any check fails, so it also works in a script. Useful for auditing work that predates the
|
||||
pipeline, and for proving the checks on real shots before turning the stage on.
|
||||
|
||||
Preview generation (Phase 4) plugs into this same service.
|
||||
|
||||
## Packaging for another machine
|
||||
|
||||
@@ -117,6 +156,7 @@ Create `C:\ProgramData\VFXReviewWorker\config.json` (§7.8):
|
||||
"aerenderPath": "C:\\Program Files\\Adobe\\Adobe After Effects 2026\\Support Files\\aerender.exe",
|
||||
"aeVersion": "24.3",
|
||||
"ffmpegPath": "C:\\pipeline\\bin\\ffmpeg.exe",
|
||||
"plateSearchRoots": ["V:/_FOOTAGE/UNG"],
|
||||
"pathMappings": [
|
||||
{ "from": "//SAN/", "to": "S:/" }
|
||||
]
|
||||
@@ -124,7 +164,9 @@ Create `C:\ProgramData\VFXReviewWorker\config.json` (§7.8):
|
||||
```
|
||||
|
||||
`pathMappings` translate the manifest's canonical UNC paths to this machine's drive
|
||||
mappings. Everything tunable (poll interval, lease, stall timeout, max attempts) lives in
|
||||
mappings. `plateSearchRoots` is optional: it is only consulted when a manifest carries no
|
||||
timing plate (older panel versions), to find the shot's plates for the timecode check.
|
||||
Everything tunable (poll interval, lease, stall timeout, max attempts, validation) lives in
|
||||
the server's SystemConfig and arrives at registration — no per-machine tuning files.
|
||||
|
||||
## Run interactively (first-time smoke test)
|
||||
|
||||
@@ -0,0 +1,177 @@
|
||||
using Microsoft.Extensions.Logging.Abstractions;
|
||||
using VFXReviewWorker;
|
||||
using Xunit;
|
||||
|
||||
namespace VFXReviewWorker.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// Packaging is the last step before files leave the studio, so the parts that
|
||||
/// could silently ship the wrong thing — checksum verification, frame-count
|
||||
/// drift, hard-link vs copy — are pinned here.
|
||||
/// </summary>
|
||||
public class DeliveryStageTests : IDisposable
|
||||
{
|
||||
private readonly string _root = Path.Combine(Path.GetTempPath(), $"vfxr_delivery_{Guid.NewGuid():N}");
|
||||
private readonly string _sourceDir;
|
||||
private readonly string _packageDir;
|
||||
|
||||
public DeliveryStageTests()
|
||||
{
|
||||
_sourceDir = Path.Combine(_root, "renders", "SHOT_001", "v001");
|
||||
_packageDir = Path.Combine(_root, "DELIVERY", "101", "260813_Delivery");
|
||||
Directory.CreateDirectory(_sourceDir);
|
||||
}
|
||||
|
||||
public void Dispose()
|
||||
{
|
||||
try { Directory.Delete(_root, recursive: true); } catch { }
|
||||
}
|
||||
|
||||
private void WriteFrames(int count, int startFrame = 1001)
|
||||
{
|
||||
for (var i = 0; i < count; i++)
|
||||
{
|
||||
var name = $"SHOT_001_cmp_TT_v001.{startFrame + i:D5}.exr";
|
||||
File.WriteAllText(Path.Combine(_sourceDir, name), $"frame {startFrame + i} pixels");
|
||||
}
|
||||
}
|
||||
|
||||
private DeliveryManifest Manifest(string? checksum = null, int fileCount = 3, string copyMode = "auto") => new()
|
||||
{
|
||||
PackageId = "pkg1",
|
||||
PackageName = "260813_Delivery",
|
||||
PackageNumber = 1,
|
||||
Episode = "101",
|
||||
PackageDir = _packageDir,
|
||||
ReportFile = Path.Combine(_packageDir, "260813_Delivery_report.csv"),
|
||||
CopyMode = copyMode,
|
||||
Items = new List<DeliveryManifestItem>
|
||||
{
|
||||
new()
|
||||
{
|
||||
ItemId = "item1",
|
||||
ExportId = "exp1",
|
||||
ShotCode = "SHOT_001",
|
||||
VersionString = "v001",
|
||||
SourceDir = _sourceDir,
|
||||
OutputPattern = "SHOT_001_cmp_TT_v001.[#####].exr",
|
||||
FrameStart = 1001,
|
||||
FrameEnd = 1000 + fileCount,
|
||||
FileCount = fileCount,
|
||||
Checksum = checksum,
|
||||
ShotDir = Path.Combine(_packageDir, "SHOT_001"),
|
||||
},
|
||||
},
|
||||
};
|
||||
|
||||
private DeliveryStage NewStage() => new(
|
||||
new WorkerOptions { ServerUrl = "n/a", ApiKey = "n/a" },
|
||||
new ApiClient(new WorkerOptions { ServerUrl = "http://localhost:1", ApiKey = "x" }, NullLogger<ApiClient>.Instance),
|
||||
NullLogger<DeliveryStage>.Instance);
|
||||
|
||||
[Fact]
|
||||
public async Task PackagesFramesAndWritesReport()
|
||||
{
|
||||
WriteFrames(3);
|
||||
var result = await NewStage().RunAsync(new ClaimedJob { Id = "job1" }, Manifest(), "machine1",
|
||||
CancellationToken.None, CancellationToken.None);
|
||||
|
||||
Assert.True(result.Ok);
|
||||
var item = Assert.Single(result.Items);
|
||||
Assert.Equal("VERIFIED", item.Status);
|
||||
Assert.Equal(3, item.FileCount);
|
||||
|
||||
var delivered = Directory.GetFiles(Path.Combine(_packageDir, "SHOT_001"), "*.exr");
|
||||
Assert.Equal(3, delivered.Length);
|
||||
Assert.True(File.Exists(Path.Combine(_packageDir, "manifest.json")));
|
||||
|
||||
var report = File.ReadAllText(Path.Combine(_packageDir, "260813_Delivery_report.csv"));
|
||||
Assert.Contains("Shot,Version,FrameStart", report);
|
||||
Assert.Contains("SHOT_001,v001,1001,1003,3,3", report);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task HardLinksOnTheSameVolumeSoNoExtraBytesAreWritten()
|
||||
{
|
||||
WriteFrames(2);
|
||||
var result = await NewStage().RunAsync(new ClaimedJob { Id = "job1" }, Manifest(fileCount: 2), "machine1",
|
||||
CancellationToken.None, CancellationToken.None);
|
||||
|
||||
Assert.Equal("hardlink", result.Items[0].LinkMode);
|
||||
|
||||
// A hard link is the same file: writing through one name is visible via
|
||||
// the other. That is what makes packaging near-instant and free.
|
||||
var source = Path.Combine(_sourceDir, "SHOT_001_cmp_TT_v001.01001.exr");
|
||||
var linked = Path.Combine(_packageDir, "SHOT_001", "SHOT_001_cmp_TT_v001.01001.exr");
|
||||
File.WriteAllText(source, "rewritten");
|
||||
Assert.Equal("rewritten", File.ReadAllText(linked));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task AlwaysCopyModeProducesIndependentFiles()
|
||||
{
|
||||
WriteFrames(2);
|
||||
var result = await NewStage().RunAsync(new ClaimedJob { Id = "job1" }, Manifest(fileCount: 2, copyMode: "always-copy"),
|
||||
"machine1", CancellationToken.None, CancellationToken.None);
|
||||
|
||||
Assert.Equal("copy", result.Items[0].LinkMode);
|
||||
|
||||
var source = Path.Combine(_sourceDir, "SHOT_001_cmp_TT_v001.01001.exr");
|
||||
var copied = Path.Combine(_packageDir, "SHOT_001", "SHOT_001_cmp_TT_v001.01001.exr");
|
||||
File.WriteAllText(source, "rewritten");
|
||||
Assert.NotEqual("rewritten", File.ReadAllText(copied));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task ChecksumMismatchFailsTheItem()
|
||||
{
|
||||
WriteFrames(3);
|
||||
var result = await NewStage().RunAsync(new ClaimedJob { Id = "job1" },
|
||||
Manifest(checksum: "deadbeefdeadbeef"), "machine1", CancellationToken.None, CancellationToken.None);
|
||||
|
||||
Assert.False(result.Ok);
|
||||
Assert.Equal("FAILED", result.Items[0].Status);
|
||||
Assert.Contains("Checksum mismatch", result.Items[0].ErrorMessage!);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task FrameCountDriftSinceQcFailsTheItem()
|
||||
{
|
||||
// The export recorded 3 frames at validation; the folder now has 2.
|
||||
WriteFrames(2);
|
||||
var result = await NewStage().RunAsync(new ClaimedJob { Id = "job1" }, Manifest(fileCount: 3), "machine1",
|
||||
CancellationToken.None, CancellationToken.None);
|
||||
|
||||
Assert.False(result.Ok);
|
||||
Assert.Contains("the render changed since QC", result.Items[0].ErrorMessage!);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task MissingSourceFolderFailsWithThePath()
|
||||
{
|
||||
var manifest = Manifest();
|
||||
manifest.Items[0].SourceDir = Path.Combine(_root, "does-not-exist");
|
||||
|
||||
var result = await NewStage().RunAsync(new ClaimedJob { Id = "job1" }, manifest, "machine1",
|
||||
CancellationToken.None, CancellationToken.None);
|
||||
|
||||
Assert.False(result.Ok);
|
||||
Assert.Contains("Source folder not found", result.Items[0].ErrorMessage!);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task ChecksumMatchesTheValidationTimeDigestForIdenticalContent()
|
||||
{
|
||||
WriteFrames(3);
|
||||
var first = await NewStage().RunAsync(new ClaimedJob { Id = "job1" }, Manifest(), "machine1",
|
||||
CancellationToken.None, CancellationToken.None);
|
||||
|
||||
// Feeding the recorded digest back in must verify, which is exactly what
|
||||
// the delivery builder does against Export.checksum.
|
||||
Directory.Delete(_packageDir, recursive: true);
|
||||
var second = await NewStage().RunAsync(new ClaimedJob { Id = "job1" },
|
||||
Manifest(checksum: first.Items[0].Checksum), "machine1", CancellationToken.None, CancellationToken.None);
|
||||
|
||||
Assert.True(second.Ok);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,193 @@
|
||||
using VFXReviewWorker;
|
||||
using Xunit;
|
||||
|
||||
namespace VFXReviewWorker.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// The timecode check is the reason the validation engine exists (§8.1a), so
|
||||
/// the header parse and the timecode arithmetic are pinned here against a
|
||||
/// synthetic EXR built byte by byte.
|
||||
/// </summary>
|
||||
public class ExrHeaderTests
|
||||
{
|
||||
[Fact]
|
||||
public void ReadsTimecodeResolutionChannelsAndFps()
|
||||
{
|
||||
var path = ExrFixture.Write(timecode: "18:58:57:12", width: 1920, height: 1080,
|
||||
channels: new[] { ("A", 1), ("B", 1), ("G", 1), ("R", 1) }, fpsNum: 24, compression: 4);
|
||||
try
|
||||
{
|
||||
var header = ExrHeader.Read(path);
|
||||
|
||||
Assert.Equal(new Timecode(18, 58, 57, 12, false), header.Timecode);
|
||||
Assert.Equal(1920, header.DataWidth);
|
||||
Assert.Equal(1080, header.DataHeight);
|
||||
Assert.Equal(24, header.FramesPerSecond);
|
||||
Assert.Equal("half", header.BitDepth);
|
||||
Assert.True(header.HasAlpha);
|
||||
Assert.Equal("PIZ", header.CompressionName);
|
||||
}
|
||||
finally { File.Delete(path); }
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void MissingTimecodeIsReportedAsAbsentNotAsZero()
|
||||
{
|
||||
var path = ExrFixture.Write(timecode: null, width: 1920, height: 1080,
|
||||
channels: new[] { ("B", 1), ("G", 1), ("R", 1) }, fpsNum: 24, compression: 3);
|
||||
try
|
||||
{
|
||||
var header = ExrHeader.Read(path);
|
||||
Assert.Null(header.Timecode);
|
||||
Assert.False(header.HasAlpha);
|
||||
}
|
||||
finally { File.Delete(path); }
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void NonExrFileThrows()
|
||||
{
|
||||
var path = Path.GetTempFileName();
|
||||
try
|
||||
{
|
||||
File.WriteAllText(path, "this is not an exr at all, not even close");
|
||||
Assert.Throws<InvalidDataException>(() => ExrHeader.Read(path));
|
||||
Assert.Null(ExrHeader.TryRead(path));
|
||||
}
|
||||
finally { File.Delete(path); }
|
||||
}
|
||||
|
||||
[Theory]
|
||||
// The BCD packing means the raw value reads back as its own hex digits.
|
||||
[InlineData(0x18585712u, 18, 58, 57, 12)]
|
||||
[InlineData(0x00004117u, 0, 0, 41, 17)]
|
||||
[InlineData(0x00000000u, 0, 0, 0, 0)]
|
||||
public void DecodesPackedSmpteTimecode(uint packed, int hh, int mm, int ss, int ff)
|
||||
{
|
||||
Assert.Equal(new Timecode(hh, mm, ss, ff, false), Timecode.FromPacked(packed));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void FrameArithmeticRoundTrips()
|
||||
{
|
||||
var tc = new Timecode(18, 58, 57, 12, false);
|
||||
Assert.Equal(tc, Timecode.FromFrames(tc.ToFrames(24), 24));
|
||||
|
||||
// 133 frames on from 18:58:57:12 at 24 fps is 18:59:03:01 — the real
|
||||
// first/last frame pair measured on UNG_102_019_010 v005.
|
||||
Assert.Equal(new Timecode(18, 59, 3, 1, false), Timecode.FromFrames(tc.ToFrames(24) + 133, 24));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ApiTimecodeIsOneHandleAheadOfThePlate()
|
||||
{
|
||||
// The defect class from §8.1a: the API stores the editorial in-point and
|
||||
// the plate starts 8 frames earlier, so a comp anchored to the API value
|
||||
// renders 8 frames late.
|
||||
var plate = Timecode.Parse("13:44:06:02")!.Value;
|
||||
var api = Timecode.Parse("13:44:06:10")!.Value;
|
||||
Assert.Equal(8, api.ToFrames(24) - plate.ToFrames(24));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void FrameDerivedTimecodeIsFrameNumberOverFps()
|
||||
{
|
||||
// 00:00:41:17 is frame 1001 at 24 fps — the signature of a comp built
|
||||
// with no source timecode at all.
|
||||
Assert.Equal(1001, Timecode.Parse("00:00:41:17")!.Value.ToFrames(24));
|
||||
}
|
||||
|
||||
[Theory]
|
||||
[InlineData("UNG_102_019_010_cmp_TT_v005.01001.exr", 1001)]
|
||||
[InlineData("UNG_108_030_010_cmp_TT_v003.01204.exr", 1204)]
|
||||
[InlineData("no_frame_number.exr", null)]
|
||||
public void ParsesFrameNumbersFromFilenames(string name, int? expected)
|
||||
{
|
||||
Assert.Equal(expected, SequencePattern.FrameFromName(name));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ParsesOutputPattern()
|
||||
{
|
||||
var p = SequencePattern.FromOutputPattern("UNG_102_019_010_cmp_TT_v005.[#####].exr");
|
||||
Assert.Equal("UNG_102_019_010_cmp_TT_v005", p.Prefix);
|
||||
Assert.Equal(".exr", p.Extension);
|
||||
Assert.Equal(5, p.Padding);
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>Builds a minimal but structurally real EXR header on disk.</summary>
|
||||
internal static class ExrFixture
|
||||
{
|
||||
public static string Write(string? timecode, int width, int height,
|
||||
(string Name, int PixelType)[] channels, int fpsNum, byte compression)
|
||||
{
|
||||
var path = Path.Combine(Path.GetTempPath(), $"vfxr_test_{Guid.NewGuid():N}.exr");
|
||||
using var fs = new FileStream(path, FileMode.Create, FileAccess.Write);
|
||||
using var w = new BinaryWriter(fs);
|
||||
|
||||
w.Write(ExrHeader.Magic);
|
||||
w.Write(2); // version 2, no flags
|
||||
|
||||
// channels (chlist)
|
||||
using (var chData = new MemoryStream())
|
||||
using (var cw = new BinaryWriter(chData))
|
||||
{
|
||||
foreach (var (name, pixelType) in channels)
|
||||
{
|
||||
WriteCString(cw, name);
|
||||
cw.Write(pixelType);
|
||||
cw.Write((byte)0); // pLinear
|
||||
cw.Write(new byte[3]); // reserved
|
||||
cw.Write(1); // xSampling
|
||||
cw.Write(1); // ySampling
|
||||
}
|
||||
cw.Write((byte)0); // list terminator
|
||||
WriteAttr(w, "channels", "chlist", chData.ToArray());
|
||||
}
|
||||
|
||||
WriteAttr(w, "compression", "compression", new[] { compression });
|
||||
|
||||
var window = new byte[16];
|
||||
BitConverter.GetBytes(0).CopyTo(window, 0);
|
||||
BitConverter.GetBytes(0).CopyTo(window, 4);
|
||||
BitConverter.GetBytes(width - 1).CopyTo(window, 8);
|
||||
BitConverter.GetBytes(height - 1).CopyTo(window, 12);
|
||||
WriteAttr(w, "dataWindow", "box2i", window);
|
||||
WriteAttr(w, "displayWindow", "box2i", window);
|
||||
|
||||
var rational = new byte[8];
|
||||
BitConverter.GetBytes(fpsNum).CopyTo(rational, 0);
|
||||
BitConverter.GetBytes(1u).CopyTo(rational, 4);
|
||||
WriteAttr(w, "framesPerSecond", "rational", rational);
|
||||
|
||||
if (timecode is not null)
|
||||
{
|
||||
var parts = timecode.Split(':');
|
||||
uint Bcd(int v) => (uint)((v / 10) * 16 + v % 10);
|
||||
var packed = (Bcd(int.Parse(parts[0])) << 24) | (Bcd(int.Parse(parts[1])) << 16)
|
||||
| (Bcd(int.Parse(parts[2])) << 8) | Bcd(int.Parse(parts[3]));
|
||||
var tcData = new byte[8];
|
||||
BitConverter.GetBytes(packed).CopyTo(tcData, 0);
|
||||
WriteAttr(w, "timeCode", "timecode", tcData);
|
||||
WriteAttr(w, "timeCodeString", "string", System.Text.Encoding.ASCII.GetBytes(timecode));
|
||||
}
|
||||
|
||||
w.Write((byte)0); // end of header
|
||||
return path;
|
||||
}
|
||||
|
||||
private static void WriteAttr(BinaryWriter w, string name, string type, byte[] data)
|
||||
{
|
||||
WriteCString(w, name);
|
||||
WriteCString(w, type);
|
||||
w.Write(data.Length);
|
||||
w.Write(data);
|
||||
}
|
||||
|
||||
private static void WriteCString(BinaryWriter w, string s)
|
||||
{
|
||||
w.Write(System.Text.Encoding.ASCII.GetBytes(s));
|
||||
w.Write((byte)0);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,61 @@
|
||||
using System.Text;
|
||||
using VFXReviewWorker;
|
||||
using Xunit;
|
||||
|
||||
namespace VFXReviewWorker.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// Reference vectors from the xxHash specification. The delivery builder
|
||||
/// verifies packaged files against digests recorded here at validation time, so
|
||||
/// a wrong implementation would not be caught by round-tripping our own code —
|
||||
/// only by pinning the published values.
|
||||
/// </summary>
|
||||
public class XxHash64Tests
|
||||
{
|
||||
[Theory]
|
||||
[InlineData("", 0xEF46DB3751D8E999UL)]
|
||||
[InlineData("a", 0xD24EC4F1A98C6E5BUL)]
|
||||
[InlineData("abc", 0x44BC2CF5AD770999UL)]
|
||||
public void MatchesReferenceVectors(string input, ulong expected)
|
||||
{
|
||||
Assert.Equal(expected, XxHash64.Hash(Encoding.ASCII.GetBytes(input)));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void StreamingInChunksMatchesSingleShot()
|
||||
{
|
||||
var data = new byte[1000];
|
||||
new Random(42).NextBytes(data);
|
||||
var oneShot = XxHash64.Hash(data);
|
||||
|
||||
// Chunk sizes deliberately straddle the 32-byte stripe boundary.
|
||||
foreach (var chunk in new[] { 1, 7, 31, 32, 33, 64, 999 })
|
||||
{
|
||||
var state = new XxHash64.State(0);
|
||||
for (var i = 0; i < data.Length; i += chunk)
|
||||
{
|
||||
state.Update(new ReadOnlySpan<byte>(data, i, Math.Min(chunk, data.Length - i)));
|
||||
}
|
||||
Assert.Equal(oneShot, state.Digest());
|
||||
}
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void HashFileIsStableAndContentSensitive()
|
||||
{
|
||||
var path = Path.GetTempFileName();
|
||||
try
|
||||
{
|
||||
File.WriteAllText(path, "frame data");
|
||||
var first = XxHash64.HashFile(path);
|
||||
Assert.Equal(first, XxHash64.HashFile(path));
|
||||
|
||||
File.WriteAllText(path, "frame datb");
|
||||
Assert.NotEqual(first, XxHash64.HashFile(path));
|
||||
}
|
||||
finally
|
||||
{
|
||||
File.Delete(path);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -71,17 +71,32 @@ public sealed class ApiClient
|
||||
return body?.Job;
|
||||
}
|
||||
|
||||
// E9 — best-effort; failures are swallowed by the caller (lease/reaper covers us)
|
||||
// E9 — best-effort by contract: a progress report is a nicety, and losing
|
||||
// one must never abort work in flight. Transport failures return null here
|
||||
// rather than throwing at the call site; the lease and the reaper are what
|
||||
// actually protect the job.
|
||||
public async Task<ProgressResponse?> ProgressAsync(string jobId, string machineId, double progress,
|
||||
int? currentFrame, int? totalFrames, int? etaSeconds, string? logTail, CancellationToken ct)
|
||||
{
|
||||
using var req = new HttpRequestMessage(HttpMethod.Patch, $"api/ext/render/jobs/{jobId}/progress")
|
||||
try
|
||||
{
|
||||
Content = JsonContent.Create(new { machineId, progress, currentFrame, totalFrames, etaSeconds, logTail }, options: Json),
|
||||
};
|
||||
var res = await _http.SendAsync(req, ct);
|
||||
if (!res.IsSuccessStatusCode) return null;
|
||||
return await res.Content.ReadFromJsonAsync<ProgressResponse>(Json, ct);
|
||||
using var req = new HttpRequestMessage(HttpMethod.Patch, $"api/ext/render/jobs/{jobId}/progress")
|
||||
{
|
||||
Content = JsonContent.Create(new { machineId, progress, currentFrame, totalFrames, etaSeconds, logTail }, options: Json),
|
||||
};
|
||||
var res = await _http.SendAsync(req, ct);
|
||||
if (!res.IsSuccessStatusCode) return null;
|
||||
return await res.Content.ReadFromJsonAsync<ProgressResponse>(Json, ct);
|
||||
}
|
||||
catch (OperationCanceledException) when (ct.IsCancellationRequested)
|
||||
{
|
||||
throw; // a real shutdown still stops us
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_log.LogDebug(ex, "Progress report failed for job {JobId} (non-fatal)", jobId);
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
// E10 / E11 — lifecycle reports, sent via the spooler for durability.
|
||||
@@ -93,6 +108,8 @@ public sealed class ApiClient
|
||||
{
|
||||
"fail" => $"api/ext/render/jobs/{jobId}/fail",
|
||||
"complete-render" => $"api/ext/render/jobs/{jobId}/complete-render",
|
||||
"validation" => $"api/ext/render/jobs/{jobId}/validation",
|
||||
"delivery" => $"api/ext/render/jobs/{jobId}/delivery",
|
||||
"finalize" => $"api/ext/render/jobs/{jobId}/finalize",
|
||||
_ => throw new ArgumentOutOfRangeException(nameof(kind), kind, "unknown lifecycle report kind"),
|
||||
};
|
||||
|
||||
@@ -0,0 +1,327 @@
|
||||
using System.Globalization;
|
||||
using System.Runtime.InteropServices;
|
||||
using System.Text;
|
||||
using System.Text.Json;
|
||||
using Microsoft.Extensions.Logging;
|
||||
|
||||
namespace VFXReviewWorker;
|
||||
|
||||
/// <summary>
|
||||
/// Delivery builder (RenderPipeline2 §11). Creates the dated package folder,
|
||||
/// hard-links (or copies) each shot's EXRs and delivery MOV into it, verifies
|
||||
/// what it wrote, and leaves a manifest + CSV report beside the shots.
|
||||
///
|
||||
/// Hard links make an 18-shot EXR package near-instant and cost no extra disk.
|
||||
/// They are safe here because rendered versions are immutable by pipeline
|
||||
/// design — a new version is a new directory, nothing ever rewrites an EXR in
|
||||
/// place. Cross-volume targets fall back to copying automatically.
|
||||
/// </summary>
|
||||
public sealed class DeliveryStage
|
||||
{
|
||||
private readonly WorkerOptions _options;
|
||||
private readonly ApiClient _api;
|
||||
private readonly ILogger<DeliveryStage> _log;
|
||||
|
||||
public DeliveryStage(WorkerOptions options, ApiClient api, ILogger<DeliveryStage> log)
|
||||
{
|
||||
_options = options;
|
||||
_api = api;
|
||||
_log = log;
|
||||
}
|
||||
|
||||
public async Task<DeliveryResult> RunAsync(
|
||||
ClaimedJob job, DeliveryManifest manifest, string machineId, CancellationToken cancelJob, CancellationToken shutdown)
|
||||
{
|
||||
var mapper = new PathMapper(_options.PathMappings);
|
||||
var packageDir = mapper.Map(manifest.PackageDir);
|
||||
var results = new List<DeliveryItemResult>();
|
||||
var manifestFiles = new List<object>();
|
||||
var alwaysCopy = string.Equals(manifest.CopyMode, "always-copy", StringComparison.OrdinalIgnoreCase);
|
||||
|
||||
_log.LogInformation("Building delivery package {Name} ({Count} shots) at {Dir}",
|
||||
manifest.PackageName, manifest.Items.Count, packageDir);
|
||||
Directory.CreateDirectory(packageDir);
|
||||
|
||||
for (var i = 0; i < manifest.Items.Count; i++)
|
||||
{
|
||||
if (cancelJob.IsCancellationRequested) break;
|
||||
var item = manifest.Items[i];
|
||||
|
||||
await _api.ProgressAsync(job.Id, machineId, (double)i / manifest.Items.Count, i, manifest.Items.Count, null,
|
||||
$"Packaging {item.ShotCode} {item.VersionString} ({i + 1}/{manifest.Items.Count})", shutdown);
|
||||
|
||||
var result = BuildItem(item, mapper, alwaysCopy, manifestFiles);
|
||||
results.Add(result);
|
||||
if (result.Status == "FAILED")
|
||||
{
|
||||
_log.LogError("Delivery item {Shot} failed: {Error}", item.ShotCode, result.ErrorMessage);
|
||||
}
|
||||
}
|
||||
|
||||
var ok = results.Count == manifest.Items.Count && results.All(r => r.Status == "VERIFIED");
|
||||
|
||||
// The report is the client-facing document, so it is written even for a
|
||||
// partial build — it is the fastest way to see what did and did not land.
|
||||
string? reportPath = null;
|
||||
try
|
||||
{
|
||||
reportPath = WriteReport(manifest, results, mapper);
|
||||
WriteManifestJson(manifest, results, manifestFiles, packageDir);
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_log.LogWarning(ex, "Could not write delivery report/manifest for {Name}", manifest.PackageName);
|
||||
}
|
||||
|
||||
return new DeliveryResult(ok, results, packageDir, reportPath, manifestFiles);
|
||||
}
|
||||
|
||||
private DeliveryItemResult BuildItem(
|
||||
DeliveryManifestItem item, PathMapper mapper, bool alwaysCopy, List<object> manifestFiles)
|
||||
{
|
||||
var sourceDir = mapper.Map(item.SourceDir);
|
||||
var shotDir = mapper.Map(item.ShotDir);
|
||||
|
||||
try
|
||||
{
|
||||
if (!Directory.Exists(sourceDir))
|
||||
{
|
||||
return DeliveryItemResult.Failed(item, $"Source folder not found: {sourceDir}");
|
||||
}
|
||||
|
||||
var pattern = SequencePattern.FromOutputPattern(item.OutputPattern);
|
||||
var sources = Directory.EnumerateFiles(sourceDir)
|
||||
.Where(f => Path.GetFileName(f).StartsWith(pattern.Prefix, StringComparison.OrdinalIgnoreCase)
|
||||
&& Path.GetFileName(f).EndsWith(pattern.Extension, StringComparison.OrdinalIgnoreCase))
|
||||
.OrderBy(f => f, StringComparer.Ordinal)
|
||||
.ToList();
|
||||
|
||||
if (sources.Count == 0)
|
||||
{
|
||||
return DeliveryItemResult.Failed(item, $"No {pattern.Extension} files matching {pattern.Prefix}* in {sourceDir}");
|
||||
}
|
||||
if (item.FileCount is int expected && sources.Count != expected)
|
||||
{
|
||||
return DeliveryItemResult.Failed(item,
|
||||
$"Source has {sources.Count} frames but the export recorded {expected} — the render changed since QC");
|
||||
}
|
||||
|
||||
Directory.CreateDirectory(shotDir);
|
||||
|
||||
long totalBytes = 0;
|
||||
var usedCopy = false;
|
||||
foreach (var src in sources)
|
||||
{
|
||||
var dest = Path.Combine(shotDir, Path.GetFileName(src));
|
||||
if (Place(src, dest, alwaysCopy, out var copied) is string placeError)
|
||||
{
|
||||
return DeliveryItemResult.Failed(item, placeError);
|
||||
}
|
||||
usedCopy |= copied;
|
||||
totalBytes += new FileInfo(dest).Length;
|
||||
}
|
||||
|
||||
// Verification reads the delivered bytes back, which works the same
|
||||
// whether the entry is a link or a copy — `fsutil hardlink list` is
|
||||
// not available over SMB, so link identity cannot be queried.
|
||||
var digest = SequenceChecksum(sources.Select(s => Path.Combine(shotDir, Path.GetFileName(s))));
|
||||
if (item.Checksum is { Length: > 0 } && !string.Equals(digest, item.Checksum, StringComparison.OrdinalIgnoreCase))
|
||||
{
|
||||
return DeliveryItemResult.Failed(item,
|
||||
$"Checksum mismatch after packaging: expected {item.Checksum}, got {digest}");
|
||||
}
|
||||
|
||||
string? movDest = null;
|
||||
if (!string.IsNullOrWhiteSpace(item.DeliveryMovPath))
|
||||
{
|
||||
var movSrc = mapper.Map(item.DeliveryMovPath!);
|
||||
if (File.Exists(movSrc))
|
||||
{
|
||||
movDest = Path.Combine(shotDir, Path.GetFileName(movSrc));
|
||||
if (Place(movSrc, movDest, alwaysCopy, out var movCopied) is string movError)
|
||||
{
|
||||
return DeliveryItemResult.Failed(item, movError);
|
||||
}
|
||||
usedCopy |= movCopied;
|
||||
totalBytes += new FileInfo(movDest).Length;
|
||||
}
|
||||
else
|
||||
{
|
||||
// Not fatal: the EXRs are the deliverable, the MOV is a
|
||||
// viewing copy. It is recorded so it is not silently absent.
|
||||
_log.LogWarning("Delivery MOV missing for {Shot}: {Path}", item.ShotCode, movSrc);
|
||||
}
|
||||
}
|
||||
|
||||
foreach (var src in sources)
|
||||
{
|
||||
var name = Path.GetFileName(src);
|
||||
manifestFiles.Add(new
|
||||
{
|
||||
shotCode = item.ShotCode,
|
||||
relativePath = Path.Combine(Path.GetFileName(shotDir), name).Replace('\\', '/'),
|
||||
bytes = new FileInfo(Path.Combine(shotDir, name)).Length,
|
||||
exportId = item.ExportId,
|
||||
});
|
||||
}
|
||||
|
||||
return new DeliveryItemResult
|
||||
{
|
||||
ItemId = item.ItemId,
|
||||
ShotCode = item.ShotCode,
|
||||
VersionString = item.VersionString,
|
||||
Status = "VERIFIED",
|
||||
ExrPath = shotDir,
|
||||
MovPath = movDest,
|
||||
FileCount = sources.Count,
|
||||
TotalBytes = totalBytes,
|
||||
Checksum = digest,
|
||||
LinkMode = usedCopy ? "copy" : "hardlink",
|
||||
FrameStart = item.FrameStart,
|
||||
FrameEnd = item.FrameEnd,
|
||||
};
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
return DeliveryItemResult.Failed(item, $"{ex.GetType().Name}: {ex.Message}");
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Hard-link when possible, copy when not. Returns null on success or the
|
||||
/// error message. `copied` reports which happened, so the package records
|
||||
/// what it actually did rather than what it intended.
|
||||
/// </summary>
|
||||
private string? Place(string source, string dest, bool alwaysCopy, out bool copied)
|
||||
{
|
||||
copied = false;
|
||||
try
|
||||
{
|
||||
if (File.Exists(dest)) File.Delete(dest);
|
||||
|
||||
if (!alwaysCopy && CreateHardLink(dest, source, IntPtr.Zero))
|
||||
{
|
||||
return null;
|
||||
}
|
||||
|
||||
// Cross-volume, a filesystem without link support, or always-copy.
|
||||
File.Copy(source, dest, overwrite: true);
|
||||
copied = true;
|
||||
return null;
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
return $"Could not place {Path.GetFileName(source)}: {ex.Message}";
|
||||
}
|
||||
}
|
||||
|
||||
private static string SequenceChecksum(IEnumerable<string> files)
|
||||
{
|
||||
var state = new XxHash64.State(0);
|
||||
foreach (var f in files)
|
||||
{
|
||||
state.Update(Encoding.ASCII.GetBytes(XxHash64.HashFile(f)));
|
||||
}
|
||||
return state.Digest().ToString("x16");
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// The client-facing delivery report (§11.4): one row per shot. Columns are
|
||||
/// confirmed against the production's delivery spec before the first real
|
||||
/// delivery (18.2-C2), which is why this is a plain, easily-edited writer.
|
||||
/// </summary>
|
||||
private string WriteReport(DeliveryManifest manifest, List<DeliveryItemResult> results, PathMapper mapper)
|
||||
{
|
||||
var path = mapper.Map(manifest.ReportFile);
|
||||
var dir = Path.GetDirectoryName(path);
|
||||
if (dir is not null) Directory.CreateDirectory(dir);
|
||||
|
||||
var sb = new StringBuilder();
|
||||
sb.AppendLine("Shot,Version,FrameStart,FrameEnd,FrameCount,Files,Bytes,Checksum,MOV,Status");
|
||||
foreach (var r in results)
|
||||
{
|
||||
sb.AppendLine(string.Join(",", new[]
|
||||
{
|
||||
Csv(r.ShotCode),
|
||||
Csv(r.VersionString),
|
||||
r.FrameStart.ToString(CultureInfo.InvariantCulture),
|
||||
r.FrameEnd.ToString(CultureInfo.InvariantCulture),
|
||||
(r.FrameEnd - r.FrameStart + 1).ToString(CultureInfo.InvariantCulture),
|
||||
(r.FileCount ?? 0).ToString(CultureInfo.InvariantCulture),
|
||||
(r.TotalBytes ?? 0).ToString(CultureInfo.InvariantCulture),
|
||||
Csv(r.Checksum ?? ""),
|
||||
Csv(r.MovPath is null ? "" : Path.GetFileName(r.MovPath)),
|
||||
Csv(r.Status),
|
||||
}));
|
||||
}
|
||||
File.WriteAllText(path, sb.ToString(), new UTF8Encoding(false));
|
||||
_log.LogInformation("Wrote delivery report {Path}", path);
|
||||
return path;
|
||||
}
|
||||
|
||||
private void WriteManifestJson(
|
||||
DeliveryManifest manifest, List<DeliveryItemResult> results, List<object> files, string packageDir)
|
||||
{
|
||||
var payload = new
|
||||
{
|
||||
package = manifest.PackageName,
|
||||
packageNumber = manifest.PackageNumber,
|
||||
episode = manifest.Episode,
|
||||
builtAt = DateTimeOffset.UtcNow,
|
||||
builtBy = Environment.MachineName,
|
||||
checksumAlgo = manifest.ChecksumAlgo,
|
||||
shots = results.Select(r => new
|
||||
{
|
||||
r.ShotCode,
|
||||
r.VersionString,
|
||||
r.FileCount,
|
||||
r.TotalBytes,
|
||||
r.Checksum,
|
||||
r.LinkMode,
|
||||
r.Status,
|
||||
r.ErrorMessage,
|
||||
}),
|
||||
files,
|
||||
};
|
||||
var path = Path.Combine(packageDir, "manifest.json");
|
||||
File.WriteAllText(path, JsonSerializer.Serialize(payload, WorkerOptions.JsonOpts), new UTF8Encoding(false));
|
||||
}
|
||||
|
||||
private static string Csv(string value) =>
|
||||
value.Contains(',') || value.Contains('"') ? "\"" + value.Replace("\"", "\"\"") + "\"" : value;
|
||||
|
||||
[DllImport("Kernel32.dll", CharSet = CharSet.Unicode, SetLastError = true)]
|
||||
[return: MarshalAs(UnmanagedType.Bool)]
|
||||
private static extern bool CreateHardLink(string lpFileName, string lpExistingFileName, IntPtr lpSecurityAttributes);
|
||||
}
|
||||
|
||||
public sealed record DeliveryResult(
|
||||
bool Ok, List<DeliveryItemResult> Items, string PackageDir, string? ReportPath, List<object> Files);
|
||||
|
||||
public sealed class DeliveryItemResult
|
||||
{
|
||||
public string ItemId { get; set; } = "";
|
||||
public string ShotCode { get; set; } = "";
|
||||
public string VersionString { get; set; } = "";
|
||||
public string Status { get; set; } = "PENDING";
|
||||
public string? ExrPath { get; set; }
|
||||
public string? MovPath { get; set; }
|
||||
public int? FileCount { get; set; }
|
||||
public long? TotalBytes { get; set; }
|
||||
public string? Checksum { get; set; }
|
||||
public string? LinkMode { get; set; }
|
||||
public string? ErrorMessage { get; set; }
|
||||
public int FrameStart { get; set; }
|
||||
public int FrameEnd { get; set; }
|
||||
|
||||
public static DeliveryItemResult Failed(DeliveryManifestItem item, string error) => new()
|
||||
{
|
||||
ItemId = item.ItemId,
|
||||
ShotCode = item.ShotCode,
|
||||
VersionString = item.VersionString,
|
||||
Status = "FAILED",
|
||||
ErrorMessage = error,
|
||||
FrameStart = item.FrameStart,
|
||||
FrameEnd = item.FrameEnd,
|
||||
};
|
||||
}
|
||||
@@ -26,6 +26,15 @@ public sealed class ServerConfig
|
||||
public int LeaseSeconds { get; set; } = 300;
|
||||
public int MaxAttempts { get; set; } = 3;
|
||||
public int StallTimeoutSeconds { get; set; } = 600;
|
||||
|
||||
/// <summary>
|
||||
/// Validation stage (§8). The server decides whether it runs, so turning it
|
||||
/// off is a config change there rather than a redeploy of every worker.
|
||||
/// </summary>
|
||||
public bool ValidationEnabled { get; set; } = true;
|
||||
public int ValidationSampleEvery { get; set; } = 25;
|
||||
/// <summary>Head handles the plates carry; only used for the API timecode fallback (§8.1a).</summary>
|
||||
public int ValidationHandleFrames { get; set; } = 8;
|
||||
}
|
||||
|
||||
public sealed class ClaimResponse
|
||||
@@ -52,6 +61,43 @@ public sealed class ClaimedJob
|
||||
public PreviewManifest AsPreviewManifest() =>
|
||||
Manifest.Deserialize<PreviewManifest>(WorkerOptions.JsonOpts)
|
||||
?? throw new InvalidOperationException("Job manifest is not a preview manifest");
|
||||
|
||||
public DeliveryManifest AsDeliveryManifest() =>
|
||||
Manifest.Deserialize<DeliveryManifest>(WorkerOptions.JsonOpts)
|
||||
?? throw new InvalidOperationException("Job manifest is not a delivery manifest");
|
||||
}
|
||||
|
||||
/// <summary>DELIVERY_BUILD manifest (server-built, RenderPipeline2 §11).</summary>
|
||||
public sealed class DeliveryManifest
|
||||
{
|
||||
public string Stage { get; set; } = "DELIVERY";
|
||||
public string PackageId { get; set; } = "";
|
||||
public string PackageName { get; set; } = "";
|
||||
public int PackageNumber { get; set; }
|
||||
public string Episode { get; set; } = "";
|
||||
public string PackageDir { get; set; } = "";
|
||||
public string ReportFile { get; set; } = "";
|
||||
public bool IncludeMovs { get; set; } = true;
|
||||
public string CopyMode { get; set; } = "auto";
|
||||
public string ChecksumAlgo { get; set; } = "xxh64";
|
||||
public List<string> ReportFormats { get; set; } = new();
|
||||
public List<DeliveryManifestItem> Items { get; set; } = new();
|
||||
}
|
||||
|
||||
public sealed class DeliveryManifestItem
|
||||
{
|
||||
public string ItemId { get; set; } = "";
|
||||
public string ExportId { get; set; } = "";
|
||||
public string ShotCode { get; set; } = "";
|
||||
public string VersionString { get; set; } = "";
|
||||
public string SourceDir { get; set; } = "";
|
||||
public string OutputPattern { get; set; } = "";
|
||||
public int FrameStart { get; set; }
|
||||
public int FrameEnd { get; set; }
|
||||
public int? FileCount { get; set; }
|
||||
public string? Checksum { get; set; }
|
||||
public string? DeliveryMovPath { get; set; }
|
||||
public string ShotDir { get; set; } = "";
|
||||
}
|
||||
|
||||
/// <summary>PREVIEW_ONLY manifest (server-built, RenderPipeline2 §9).</summary>
|
||||
@@ -107,9 +153,10 @@ public sealed class PreviewBuildResult
|
||||
public int? Height { get; set; }
|
||||
}
|
||||
|
||||
/// <summary>The Render Manifest (§6.2) — only the fields the Phase 2 render stage needs; the rest is preserved raw.</summary>
|
||||
/// <summary>The Render Manifest (§6.2) — the fields the render and validation stages need; the rest is preserved raw.</summary>
|
||||
public sealed class RenderManifest
|
||||
{
|
||||
public string ShotCode { get; set; } = "";
|
||||
public string AepPath { get; set; } = "";
|
||||
public string CompName { get; set; } = "";
|
||||
public string OutputDir { get; set; } = "";
|
||||
@@ -118,6 +165,11 @@ public sealed class RenderManifest
|
||||
public string? RenderSettingsTemplate { get; set; }
|
||||
public int FrameStart { get; set; }
|
||||
public int FrameEnd { get; set; }
|
||||
public double Fps { get; set; } = 24;
|
||||
public int Width { get; set; }
|
||||
public int Height { get; set; }
|
||||
public ManifestExpected? Expected { get; set; }
|
||||
public ManifestSource? Source { get; set; }
|
||||
|
||||
[JsonExtensionData]
|
||||
public Dictionary<string, JsonElement>? Extra { get; set; }
|
||||
@@ -125,6 +177,35 @@ public sealed class RenderManifest
|
||||
public int TotalFrames => FrameEnd - FrameStart + 1;
|
||||
}
|
||||
|
||||
/// <summary>Per-project expectations the validation engine checks against (§8.1).</summary>
|
||||
public sealed class ManifestExpected
|
||||
{
|
||||
public string? Colorspace { get; set; }
|
||||
public string? BitDepth { get; set; }
|
||||
public string? ExrCompression { get; set; }
|
||||
public bool? Alpha { get; set; }
|
||||
/// <summary>The API's editorial in-point — the rank 3 timecode fallback only (§8.1a).</summary>
|
||||
public string? TimecodeStart { get; set; }
|
||||
}
|
||||
|
||||
/// <summary>Plate provenance resolved by the panel from the comp's footage layers (§6.2).</summary>
|
||||
public sealed class ManifestSource
|
||||
{
|
||||
public PlateRef? TimingPlate { get; set; }
|
||||
public List<PlateRef> Plates { get; set; } = new();
|
||||
public string? StartSource { get; set; }
|
||||
public int? StartFrames { get; set; }
|
||||
public int? HandleFrames { get; set; }
|
||||
}
|
||||
|
||||
public sealed class PlateRef
|
||||
{
|
||||
public string? Element { get; set; }
|
||||
public string Path { get; set; } = "";
|
||||
public int? FrameStart { get; set; }
|
||||
public int? FrameEnd { get; set; }
|
||||
}
|
||||
|
||||
public sealed class ProgressResponse
|
||||
{
|
||||
public bool Ok { get; set; }
|
||||
|
||||
@@ -0,0 +1,234 @@
|
||||
using System.Text;
|
||||
|
||||
namespace VFXReviewWorker;
|
||||
|
||||
/// <summary>
|
||||
/// Minimal OpenEXR header reader (RenderPipeline2 §8.1a).
|
||||
///
|
||||
/// Everything validation needs — timecode, fps, resolution, channel types,
|
||||
/// compression — lives in the header, so a check costs one 64 KB read and no
|
||||
/// pixel decode. That is why the engine needs no oiiotool/OpenImageIO
|
||||
/// dependency and can run on a bare workstation.
|
||||
///
|
||||
/// Header layout: magic(4) version(4) then attributes as
|
||||
/// `name\0 type\0 int32 size data`, terminated by an empty name.
|
||||
/// </summary>
|
||||
public sealed class ExrHeader
|
||||
{
|
||||
public const uint Magic = 20000630;
|
||||
private const int MaxHeaderBytes = 1 << 18; // 256 KB — far beyond any sane header
|
||||
|
||||
public int DataWidth { get; private init; }
|
||||
public int DataHeight { get; private init; }
|
||||
public int DisplayWidth { get; private init; }
|
||||
public int DisplayHeight { get; private init; }
|
||||
public IReadOnlyList<ExrChannel> Channels { get; private init; } = Array.Empty<ExrChannel>();
|
||||
public byte? CompressionCode { get; private init; }
|
||||
public Timecode? Timecode { get; private init; }
|
||||
public string? TimecodeString { get; private init; }
|
||||
public double? FramesPerSecond { get; private init; }
|
||||
public bool HasChromaticities { get; private init; }
|
||||
public bool IsTiled { get; private init; }
|
||||
public IReadOnlyList<string> AttributeNames { get; private init; } = Array.Empty<string>();
|
||||
|
||||
public bool HasAlpha => Channels.Any(c => c.Name.Equals("A", StringComparison.Ordinal));
|
||||
|
||||
/// <summary>half / float / uint — the deepest channel type present.</summary>
|
||||
public string? BitDepth
|
||||
{
|
||||
get
|
||||
{
|
||||
if (Channels.Count == 0) return null;
|
||||
if (Channels.Any(c => c.PixelType == 2)) return "float";
|
||||
if (Channels.Any(c => c.PixelType == 1)) return "half";
|
||||
return "uint";
|
||||
}
|
||||
}
|
||||
|
||||
public string CompressionName => CompressionCode switch
|
||||
{
|
||||
0 => "none", 1 => "RLE", 2 => "ZIPS", 3 => "ZIP", 4 => "PIZ",
|
||||
5 => "PXR24", 6 => "B44", 7 => "B44A", 8 => "DWAA", 9 => "DWAB",
|
||||
null => "unknown",
|
||||
_ => $"code {CompressionCode}",
|
||||
};
|
||||
|
||||
/// <summary>Reads the header, or throws <see cref="InvalidDataException"/> when the file is not a readable EXR.</summary>
|
||||
public static ExrHeader Read(string path)
|
||||
{
|
||||
using var fs = new FileStream(path, FileMode.Open, FileAccess.Read, FileShare.ReadWrite);
|
||||
var buf = new byte[Math.Min(MaxHeaderBytes, Math.Max(1024, (int)Math.Min(fs.Length, MaxHeaderBytes)))];
|
||||
var read = fs.Read(buf, 0, buf.Length);
|
||||
if (read < 16) throw new InvalidDataException("File is too small to be an EXR");
|
||||
|
||||
var span = new ReadOnlySpan<byte>(buf, 0, read);
|
||||
if (BitConverter.ToUInt32(span[..4]) != Magic) throw new InvalidDataException("Not an EXR (bad magic number)");
|
||||
var version = BitConverter.ToInt32(span.Slice(4, 4));
|
||||
|
||||
int pos = 8;
|
||||
var names = new List<string>();
|
||||
var channels = new List<ExrChannel>();
|
||||
int dataW = 0, dataH = 0, dispW = 0, dispH = 0;
|
||||
byte? compression = null;
|
||||
Timecode? tc = null;
|
||||
string? tcString = null;
|
||||
double? fps = null;
|
||||
bool chroma = false;
|
||||
|
||||
while (pos < read)
|
||||
{
|
||||
var name = ReadCString(span, ref pos);
|
||||
if (name.Length == 0) break; // end of header
|
||||
var type = ReadCString(span, ref pos);
|
||||
if (pos + 4 > read) throw new InvalidDataException("Truncated EXR header (attribute size)");
|
||||
var size = BitConverter.ToInt32(span.Slice(pos, 4));
|
||||
pos += 4;
|
||||
if (size < 0 || pos + size > read) throw new InvalidDataException($"Truncated EXR header (attribute \"{name}\")");
|
||||
var data = span.Slice(pos, size);
|
||||
names.Add(name);
|
||||
|
||||
switch (name)
|
||||
{
|
||||
case "dataWindow" when size >= 16:
|
||||
dataW = BitConverter.ToInt32(data.Slice(8, 4)) - BitConverter.ToInt32(data[..4]) + 1;
|
||||
dataH = BitConverter.ToInt32(data.Slice(12, 4)) - BitConverter.ToInt32(data.Slice(4, 4)) + 1;
|
||||
break;
|
||||
case "displayWindow" when size >= 16:
|
||||
dispW = BitConverter.ToInt32(data.Slice(8, 4)) - BitConverter.ToInt32(data[..4]) + 1;
|
||||
dispH = BitConverter.ToInt32(data.Slice(12, 4)) - BitConverter.ToInt32(data.Slice(4, 4)) + 1;
|
||||
break;
|
||||
case "compression" when size >= 1:
|
||||
compression = data[0];
|
||||
break;
|
||||
case "timeCode" when size >= 8:
|
||||
tc = VFXReviewWorker.Timecode.FromPacked(
|
||||
BitConverter.ToUInt32(data[..4]), BitConverter.ToUInt32(data.Slice(4, 4)));
|
||||
break;
|
||||
case "timeCodeString":
|
||||
tcString = Encoding.ASCII.GetString(data).TrimEnd('\0');
|
||||
break;
|
||||
case "framesPerSecond" when size >= 8:
|
||||
{
|
||||
var num = BitConverter.ToInt32(data[..4]);
|
||||
var den = BitConverter.ToUInt32(data.Slice(4, 4));
|
||||
if (den != 0) fps = num / (double)den;
|
||||
break;
|
||||
}
|
||||
case "chromaticities":
|
||||
chroma = true;
|
||||
break;
|
||||
case "channels":
|
||||
channels.AddRange(ReadChannels(data));
|
||||
break;
|
||||
}
|
||||
|
||||
pos += size;
|
||||
}
|
||||
|
||||
return new ExrHeader
|
||||
{
|
||||
DataWidth = dataW,
|
||||
DataHeight = dataH,
|
||||
DisplayWidth = dispW,
|
||||
DisplayHeight = dispH,
|
||||
Channels = channels,
|
||||
CompressionCode = compression,
|
||||
Timecode = tc,
|
||||
TimecodeString = tcString,
|
||||
FramesPerSecond = fps,
|
||||
HasChromaticities = chroma,
|
||||
IsTiled = (version & 0x200) != 0,
|
||||
AttributeNames = names,
|
||||
};
|
||||
}
|
||||
|
||||
/// <summary>Header read that returns null instead of throwing — for "is this file readable at all" checks.</summary>
|
||||
public static ExrHeader? TryRead(string path)
|
||||
{
|
||||
try { return Read(path); }
|
||||
catch { return null; }
|
||||
}
|
||||
|
||||
private static IEnumerable<ExrChannel> ReadChannels(ReadOnlySpan<byte> data)
|
||||
{
|
||||
var result = new List<ExrChannel>();
|
||||
int p = 0;
|
||||
while (p < data.Length)
|
||||
{
|
||||
var name = ReadCString(data, ref p);
|
||||
if (name.Length == 0) break; // null terminator ends the channel list
|
||||
if (p + 16 > data.Length) break;
|
||||
var pixelType = BitConverter.ToInt32(data.Slice(p, 4));
|
||||
p += 16; // pixelType(4) + pLinear(1) + reserved(3) + xSampling(4) + ySampling(4)
|
||||
result.Add(new ExrChannel(name, pixelType));
|
||||
}
|
||||
return result;
|
||||
}
|
||||
|
||||
private static string ReadCString(ReadOnlySpan<byte> span, ref int pos)
|
||||
{
|
||||
int start = pos;
|
||||
while (pos < span.Length && span[pos] != 0) pos++;
|
||||
var s = Encoding.ASCII.GetString(span[start..pos]);
|
||||
pos++; // skip the terminator
|
||||
return s;
|
||||
}
|
||||
}
|
||||
|
||||
public readonly record struct ExrChannel(string Name, int PixelType)
|
||||
{
|
||||
public string TypeName => PixelType switch { 0 => "uint", 1 => "half", 2 => "float", _ => $"type {PixelType}" };
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// SMPTE 12M timecode as stored in the EXR `timeCode` attribute: BCD-packed,
|
||||
/// which is why the raw value reads back as its own hex digits
|
||||
/// (0x18585712 → 18:58:57:12).
|
||||
/// </summary>
|
||||
public readonly record struct Timecode(int Hours, int Minutes, int Seconds, int Frames, bool DropFrame)
|
||||
{
|
||||
public static Timecode FromPacked(uint timeAndFlags, uint _userData = 0)
|
||||
{
|
||||
int Bits(int shift, int mask) => (int)((timeAndFlags >> shift) & (uint)mask);
|
||||
return new Timecode(
|
||||
Bits(24, 0xF) + Bits(28, 0x3) * 10,
|
||||
Bits(16, 0xF) + Bits(20, 0x7) * 10,
|
||||
Bits(8, 0xF) + Bits(12, 0x7) * 10,
|
||||
Bits(0, 0xF) + Bits(4, 0x3) * 10,
|
||||
Bits(6, 0x1) == 1);
|
||||
}
|
||||
|
||||
public static Timecode? Parse(string? text, bool dropFrame = false)
|
||||
{
|
||||
if (string.IsNullOrWhiteSpace(text)) return null;
|
||||
var parts = text.Trim().Split(':', ';');
|
||||
if (parts.Length < 4) return null;
|
||||
if (!int.TryParse(parts[0], out var hh) || !int.TryParse(parts[1], out var mm) ||
|
||||
!int.TryParse(parts[2], out var ss) || !int.TryParse(parts[3], out var ff)) return null;
|
||||
return new Timecode(hh, mm, ss, ff, dropFrame || text.Contains(';'));
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Non-drop frame count. The studio shoots 24 fps, so drop-frame never
|
||||
/// applies; a drop-frame flag mismatch is reported as its own check rather
|
||||
/// than silently changing this arithmetic.
|
||||
/// </summary>
|
||||
public long ToFrames(double fps)
|
||||
{
|
||||
var rate = (long)Math.Round(fps <= 0 ? 24 : fps);
|
||||
return ((Hours * 60L + Minutes) * 60L + Seconds) * rate + Frames;
|
||||
}
|
||||
|
||||
public static Timecode FromFrames(long frames, double fps)
|
||||
{
|
||||
var rate = (long)Math.Round(fps <= 0 ? 24 : fps);
|
||||
if (rate <= 0) rate = 24;
|
||||
var f = Math.Max(0, frames);
|
||||
var ff = (int)(f % rate);
|
||||
var totalSeconds = f / rate;
|
||||
return new Timecode((int)(totalSeconds / 3600), (int)(totalSeconds / 60 % 60), (int)(totalSeconds % 60), ff, false);
|
||||
}
|
||||
|
||||
public override string ToString() =>
|
||||
$"{Hours:D2}:{Minutes:D2}:{Seconds:D2}{(DropFrame ? ';' : ':')}{Frames:D2}";
|
||||
}
|
||||
@@ -14,15 +14,20 @@ public sealed class JobExecutor
|
||||
private readonly ApiClient _api;
|
||||
private readonly AerenderRunner _runner;
|
||||
private readonly PreviewStage _preview;
|
||||
private readonly ValidationStage _validation;
|
||||
private readonly DeliveryStage _delivery;
|
||||
private readonly ReportSpooler _spooler;
|
||||
private readonly ILogger<JobExecutor> _log;
|
||||
|
||||
public JobExecutor(WorkerOptions options, ApiClient api, AerenderRunner runner, PreviewStage preview, ReportSpooler spooler, ILogger<JobExecutor> log)
|
||||
public JobExecutor(WorkerOptions options, ApiClient api, AerenderRunner runner, PreviewStage preview,
|
||||
ValidationStage validation, DeliveryStage delivery, ReportSpooler spooler, ILogger<JobExecutor> log)
|
||||
{
|
||||
_options = options;
|
||||
_api = api;
|
||||
_runner = runner;
|
||||
_preview = preview;
|
||||
_validation = validation;
|
||||
_delivery = delivery;
|
||||
_spooler = spooler;
|
||||
_log = log;
|
||||
}
|
||||
@@ -35,6 +40,12 @@ public sealed class JobExecutor
|
||||
return;
|
||||
}
|
||||
|
||||
if (job.Type == "DELIVERY_BUILD")
|
||||
{
|
||||
await RunDeliveryAsync(job, machineId, cancelJob, shutdown);
|
||||
return;
|
||||
}
|
||||
|
||||
var manifest = job.AsRenderManifest();
|
||||
var mapper = new PathMapper(_options.PathMappings);
|
||||
var outputDirLocal = mapper.Map(manifest.OutputDir);
|
||||
@@ -101,6 +112,15 @@ public sealed class JobExecutor
|
||||
["exrFileCount"] = fileCount,
|
||||
["exrTotalBytes"] = totalBytes,
|
||||
});
|
||||
|
||||
// Validation runs here, while we still hold the job and the
|
||||
// files are local. It is queued behind complete-render, and the
|
||||
// spooler's in-order replay is what guarantees the server has
|
||||
// moved the Export to VALIDATING before this lands (§8).
|
||||
if (config.ValidationEnabled)
|
||||
{
|
||||
await RunValidationAsync(job, manifest, machineId, outputDirLocal, config, shutdown);
|
||||
}
|
||||
}
|
||||
else if (result.ErrorMessage == "Cancelled")
|
||||
{
|
||||
@@ -149,6 +169,140 @@ public sealed class JobExecutor
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// DELIVERY_BUILD job: build the dated package folder from already-QC'd
|
||||
/// exports. No AE involved — folders, links and verification only.
|
||||
/// </summary>
|
||||
private async Task RunDeliveryAsync(ClaimedJob job, string machineId, CancellationToken cancelJob, CancellationToken shutdown)
|
||||
{
|
||||
var manifest = job.AsDeliveryManifest();
|
||||
|
||||
WriteState(new CurrentJobState
|
||||
{
|
||||
JobId = job.Id,
|
||||
ExportId = null,
|
||||
MachineId = machineId,
|
||||
OutputDirLocal = new PathMapper(_options.PathMappings).Map(manifest.PackageDir),
|
||||
ClaimedAt = DateTimeOffset.UtcNow,
|
||||
});
|
||||
|
||||
try
|
||||
{
|
||||
var result = await _delivery.RunAsync(job, manifest, machineId, cancelJob, shutdown);
|
||||
|
||||
_log.LogInformation("Delivery {Name}: {Ok} ({Verified}/{Total} shots verified)",
|
||||
manifest.PackageName, result.Ok ? "READY" : "FAILED",
|
||||
result.Items.Count(i => i.Status == "VERIFIED"), manifest.Items.Count);
|
||||
|
||||
string? reportKey = null;
|
||||
if (result.ReportPath is not null)
|
||||
{
|
||||
reportKey = await _api.UploadArtifactAsync(job.Id, machineId, "report", result.ReportPath, "text/csv", shutdown);
|
||||
}
|
||||
|
||||
_spooler.Enqueue("delivery", job.Id, new Dictionary<string, object?>
|
||||
{
|
||||
["machineId"] = machineId,
|
||||
["ok"] = result.Ok,
|
||||
["rootPath"] = result.PackageDir,
|
||||
["reportKey"] = reportKey,
|
||||
["items"] = result.Items.Select(i => new Dictionary<string, object?>
|
||||
{
|
||||
["itemId"] = i.ItemId,
|
||||
["status"] = i.Status,
|
||||
["exrPath"] = i.ExrPath,
|
||||
["movPath"] = i.MovPath,
|
||||
["fileCount"] = i.FileCount,
|
||||
["totalBytes"] = i.TotalBytes,
|
||||
["checksum"] = i.Checksum,
|
||||
["linkMode"] = i.LinkMode,
|
||||
["errorMessage"] = i.ErrorMessage,
|
||||
}).ToList(),
|
||||
["manifest"] = new Dictionary<string, object?>
|
||||
{
|
||||
["packageDir"] = result.PackageDir,
|
||||
["reportPath"] = result.ReportPath,
|
||||
["files"] = result.Files,
|
||||
},
|
||||
["errorMessage"] = result.Ok
|
||||
? null
|
||||
: string.Join("; ", result.Items.Where(i => i.Status == "FAILED").Select(i => $"{i.ShotCode}: {i.ErrorMessage}").Take(5)),
|
||||
});
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_log.LogError(ex, "Unexpected delivery error for job {JobId}", job.Id);
|
||||
_spooler.Enqueue("delivery", job.Id, new Dictionary<string, object?>
|
||||
{
|
||||
["machineId"] = machineId,
|
||||
["ok"] = false,
|
||||
["items"] = new List<object>(),
|
||||
["errorMessage"] = $"Worker exception: {ex.Message}",
|
||||
});
|
||||
}
|
||||
finally
|
||||
{
|
||||
ClearState();
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Runs the validation engine and spools the E12 report. Deliberately
|
||||
/// non-fatal on its own errors: a crash in the checks must not lose a
|
||||
/// finished render, so it is reported as a failed validation run rather
|
||||
/// than being allowed to escape.
|
||||
/// </summary>
|
||||
private async Task RunValidationAsync(
|
||||
ClaimedJob job, RenderManifest manifest, string machineId, string outputDirLocal,
|
||||
ServerConfig config, CancellationToken shutdown)
|
||||
{
|
||||
var started = DateTimeOffset.UtcNow;
|
||||
try
|
||||
{
|
||||
// Keep the lease alive: the job stays open through validation.
|
||||
await _api.ProgressAsync(job.Id, machineId, 1.0, manifest.FrameEnd, manifest.TotalFrames, 0,
|
||||
"Validating rendered sequence…", shutdown);
|
||||
|
||||
var outcome = await Task.Run(() => _validation.Run(manifest, outputDirLocal, config), shutdown);
|
||||
var seconds = (DateTimeOffset.UtcNow - started).TotalSeconds;
|
||||
|
||||
var failed = outcome.Checks.Where(c => c.Status == "FAIL").Select(c => c.CheckName).ToList();
|
||||
if (failed.Count > 0)
|
||||
{
|
||||
_log.LogWarning("Job {JobId} validation FAILED in {Sec:F1}s: {Checks}", job.Id, seconds, string.Join(", ", failed));
|
||||
}
|
||||
else
|
||||
{
|
||||
_log.LogInformation("Job {JobId} validation passed in {Sec:F1}s ({Count} checks, {Files} files)",
|
||||
job.Id, seconds, outcome.Checks.Count, outcome.FileCount);
|
||||
}
|
||||
|
||||
_spooler.Enqueue("validation", job.Id, new Dictionary<string, object?>
|
||||
{
|
||||
["machineId"] = machineId,
|
||||
["checks"] = outcome.Checks,
|
||||
["checksum"] = outcome.Checksum,
|
||||
["checksumAlgo"] = "xxh64",
|
||||
["exrFileCount"] = outcome.FileCount,
|
||||
["exrTotalBytes"] = outcome.TotalBytes,
|
||||
["validationSeconds"] = (int)seconds,
|
||||
});
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_log.LogError(ex, "Validation engine error for job {JobId}", job.Id);
|
||||
_spooler.Enqueue("validation", job.Id, new Dictionary<string, object?>
|
||||
{
|
||||
["machineId"] = machineId,
|
||||
["checks"] = new[]
|
||||
{
|
||||
ValidationCheck.Fail("engine_error", "validation completes", "worker exception",
|
||||
$"The validation engine threw: {ex.Message}"),
|
||||
},
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// PREVIEW_ONLY job: rebuild the shot around the rendered EXRs with slate
|
||||
/// and burn-ins, render the delivery MOV + review MP4, upload the review
|
||||
|
||||
@@ -10,6 +10,13 @@ using VFXReviewWorker;
|
||||
// Run interactively for debugging: VFXReviewWorker.exe [path\to\config.json]
|
||||
// Install as a service: see RenderWorker/README.md
|
||||
|
||||
// Dry-run the validation engine over an existing render folder. No server, no
|
||||
// config, no queue — useful for checking work that predates the pipeline.
|
||||
if (args.Any(a => a.Equals("--validate", StringComparison.OrdinalIgnoreCase)))
|
||||
{
|
||||
return ValidateCommand.Run(args);
|
||||
}
|
||||
|
||||
var configPath = args.FirstOrDefault(a => !a.StartsWith('-'));
|
||||
|
||||
// Install verification on a new machine — resolves paths, checks tools and
|
||||
@@ -32,6 +39,8 @@ builder.Services.AddSingleton<ApiClient>();
|
||||
builder.Services.AddSingleton<ReportSpooler>();
|
||||
builder.Services.AddSingleton<AerenderRunner>();
|
||||
builder.Services.AddSingleton<PreviewStage>();
|
||||
builder.Services.AddSingleton<ValidationStage>();
|
||||
builder.Services.AddSingleton<DeliveryStage>();
|
||||
builder.Services.AddSingleton<JobExecutor>();
|
||||
builder.Services.AddHostedService<WorkerService>();
|
||||
|
||||
|
||||
@@ -9,6 +9,11 @@ namespace VFXReviewWorker;
|
||||
/// order with exponential backoff (1 s → 60 s). Rendering continues during
|
||||
/// server downtime; a report is deleted only once the server accepts it (or
|
||||
/// permanently rejects it as contradictory).
|
||||
///
|
||||
/// Strict ordering is load-bearing beyond durability: the validation report
|
||||
/// (E12) is only legal once complete-render (E11) has moved the Export to
|
||||
/// VALIDATING, and in-order replay is what guarantees that, with no need for
|
||||
/// the worker to wait on an ACK.
|
||||
/// </summary>
|
||||
public sealed class ReportSpooler
|
||||
{
|
||||
|
||||
@@ -0,0 +1,98 @@
|
||||
using Microsoft.Extensions.Logging.Abstractions;
|
||||
|
||||
namespace VFXReviewWorker;
|
||||
|
||||
/// <summary>
|
||||
/// `VFXReviewWorker.exe --validate <renderDir> [--plate <plateFile>] [--fps 24]`
|
||||
///
|
||||
/// Runs the validation engine (§8) over a folder of already-rendered EXRs and
|
||||
/// prints the report, without a server, a job or a queue. Two uses: proving the
|
||||
/// checks against real work before the stage goes live, and answering "is this
|
||||
/// delivery good?" for a sequence that predates the pipeline.
|
||||
/// </summary>
|
||||
public static class ValidateCommand
|
||||
{
|
||||
public static int Run(string[] args)
|
||||
{
|
||||
var dir = args.SkipWhile(a => !a.Equals("--validate", StringComparison.OrdinalIgnoreCase)).Skip(1).FirstOrDefault();
|
||||
if (string.IsNullOrWhiteSpace(dir) || dir.StartsWith('-'))
|
||||
{
|
||||
Console.Error.WriteLine("Usage: VFXReviewWorker.exe --validate <renderDir> [--plate <plateFirstFrame.exr>] [--fps 24] [--sample 25]");
|
||||
return 2;
|
||||
}
|
||||
if (!Directory.Exists(dir))
|
||||
{
|
||||
Console.Error.WriteLine($"Not a directory: {dir}");
|
||||
return 2;
|
||||
}
|
||||
|
||||
var plate = ArgValue(args, "--plate");
|
||||
var fps = double.TryParse(ArgValue(args, "--fps"), out var f) ? f : 24;
|
||||
var sample = int.TryParse(ArgValue(args, "--sample"), out var s) ? s : 25;
|
||||
|
||||
var files = Directory.EnumerateFiles(dir, "*.exr").OrderBy(x => x, StringComparer.Ordinal).ToList();
|
||||
if (files.Count == 0)
|
||||
{
|
||||
Console.Error.WriteLine($"No .exr files in {dir}");
|
||||
return 2;
|
||||
}
|
||||
|
||||
// Infer what the manifest would have said from the files themselves.
|
||||
var firstName = Path.GetFileName(files[0]);
|
||||
var frames = files.Select(x => SequencePattern.FrameFromName(Path.GetFileName(x))).Where(x => x is not null).Select(x => x!.Value).ToList();
|
||||
var prefix = firstName.Contains('.') ? firstName[..firstName.IndexOf('.')] : Path.GetFileNameWithoutExtension(firstName);
|
||||
var header = ExrHeader.TryRead(files[0]);
|
||||
|
||||
var manifest = new RenderManifest
|
||||
{
|
||||
ShotCode = prefix.Split("_cmp_")[0],
|
||||
OutputDir = dir,
|
||||
OutputPattern = $"{prefix}.[#####].exr",
|
||||
FrameStart = frames.Count > 0 ? frames.Min() : 1,
|
||||
FrameEnd = frames.Count > 0 ? frames.Max() : files.Count,
|
||||
Fps = fps,
|
||||
Width = header?.DataWidth ?? 0,
|
||||
Height = header?.DataHeight ?? 0,
|
||||
Source = plate is null ? null : new ManifestSource
|
||||
{
|
||||
TimingPlate = new PlateRef { Path = plate, FrameStart = SequencePattern.FrameFromName(Path.GetFileName(plate)) },
|
||||
StartSource = "supplied on the command line",
|
||||
},
|
||||
};
|
||||
|
||||
var options = new WorkerOptions { ServerUrl = "n/a", ApiKey = "n/a" };
|
||||
var stage = new ValidationStage(options, NullLogger<ValidationStage>.Instance);
|
||||
var outcome = stage.Run(manifest, dir, new ServerConfig { ValidationSampleEvery = sample });
|
||||
|
||||
Console.WriteLine();
|
||||
Console.WriteLine($"{files.Count} files in {dir}");
|
||||
Console.WriteLine($"inferred range {manifest.FrameStart}-{manifest.FrameEnd} at {fps} fps, pattern {manifest.OutputPattern}");
|
||||
if (plate is not null) Console.WriteLine($"timing plate {plate}");
|
||||
Console.WriteLine();
|
||||
|
||||
foreach (var c in outcome.Checks)
|
||||
{
|
||||
var mark = c.Status switch { "PASS" => " ok ", "FAIL" => " FAIL ", "WARN" => " warn ", _ => " skip " };
|
||||
Console.WriteLine($"[{mark}] {c.CheckName,-24} {Describe(c)}");
|
||||
}
|
||||
|
||||
Console.WriteLine();
|
||||
Console.WriteLine($"checksum (xxh64): {outcome.Checksum ?? "(not computed)"}");
|
||||
var failed = outcome.Checks.Count(c => c.Status == "FAIL");
|
||||
Console.WriteLine(failed == 0 ? "PASSED" : $"FAILED — {failed} check(s)");
|
||||
return failed == 0 ? 0 : 1;
|
||||
}
|
||||
|
||||
private static string Describe(ValidationCheck c)
|
||||
{
|
||||
if (!string.IsNullOrWhiteSpace(c.Message)) return c.Message!;
|
||||
if (c.Expected is null && c.Actual is null) return "";
|
||||
return c.Expected == c.Actual ? c.Actual ?? "" : $"expected {c.Expected}, got {c.Actual}";
|
||||
}
|
||||
|
||||
private static string? ArgValue(string[] args, string name)
|
||||
{
|
||||
var i = Array.FindIndex(args, a => a.Equals(name, StringComparison.OrdinalIgnoreCase));
|
||||
return i >= 0 && i + 1 < args.Length ? args[i + 1] : null;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,590 @@
|
||||
using System.Text.RegularExpressions;
|
||||
using Microsoft.Extensions.Logging;
|
||||
|
||||
namespace VFXReviewWorker;
|
||||
|
||||
/// <summary>
|
||||
/// Validation engine (RenderPipeline2 §8). Runs on the worker straight after a
|
||||
/// successful render, while it still holds the job and the files are local.
|
||||
/// It only ever *reports* — the server decides what the results mean.
|
||||
///
|
||||
/// Full-sequence checks (count, gaps, names, sizes) touch every file; header
|
||||
/// checks sample first/middle/last plus every Nth frame.
|
||||
/// </summary>
|
||||
public sealed class ValidationStage
|
||||
{
|
||||
private readonly WorkerOptions _options;
|
||||
private readonly ILogger<ValidationStage> _log;
|
||||
|
||||
public ValidationStage(WorkerOptions options, ILogger<ValidationStage> log)
|
||||
{
|
||||
_options = options;
|
||||
_log = log;
|
||||
}
|
||||
|
||||
public ValidationOutcome Run(RenderManifest manifest, string outputDirLocal, ServerConfig config)
|
||||
{
|
||||
var checks = new List<ValidationCheck>();
|
||||
var files = EnumerateSequence(manifest, outputDirLocal, checks, out var frameNumbers, out var totalBytes);
|
||||
|
||||
if (files.Count == 0)
|
||||
{
|
||||
// Nothing to inspect: the sequence checks already recorded the FAIL.
|
||||
return new ValidationOutcome(checks, 0, 0, null);
|
||||
}
|
||||
|
||||
var sampleEvery = Math.Max(1, config.ValidationSampleEvery);
|
||||
var sampled = PickSamples(files, sampleEvery);
|
||||
var headers = ReadHeaders(sampled, checks);
|
||||
|
||||
if (headers.Count > 0)
|
||||
{
|
||||
CheckResolution(manifest, headers, checks);
|
||||
CheckBitDepthAndChannels(manifest, headers, checks);
|
||||
CheckCompression(manifest, headers, checks);
|
||||
CheckTimecode(manifest, headers, frameNumbers, config, checks);
|
||||
}
|
||||
|
||||
var checksum = ComputeSequenceChecksum(files);
|
||||
return new ValidationOutcome(checks, files.Count, totalBytes, checksum);
|
||||
}
|
||||
|
||||
// ── Sequence-level checks ────────────────────────────────────────────────
|
||||
|
||||
private List<string> EnumerateSequence(
|
||||
RenderManifest manifest, string outputDirLocal, List<ValidationCheck> checks,
|
||||
out List<int> frameNumbers, out long totalBytes)
|
||||
{
|
||||
frameNumbers = new List<int>();
|
||||
totalBytes = 0;
|
||||
var expectedCount = manifest.TotalFrames;
|
||||
|
||||
if (!Directory.Exists(outputDirLocal))
|
||||
{
|
||||
checks.Add(ValidationCheck.Fail("file_count", expectedCount.ToString(), "0",
|
||||
$"Output directory does not exist on this machine: {outputDirLocal}"));
|
||||
return new List<string>();
|
||||
}
|
||||
|
||||
var pattern = SequencePattern.FromOutputPattern(manifest.OutputPattern);
|
||||
var matched = new List<(int Frame, string Path)>();
|
||||
var nonConforming = new List<string>();
|
||||
|
||||
foreach (var path in Directory.EnumerateFiles(outputDirLocal))
|
||||
{
|
||||
var name = Path.GetFileName(path);
|
||||
if (!name.EndsWith(pattern.Extension, StringComparison.OrdinalIgnoreCase)) continue;
|
||||
// Only files belonging to this render: a version folder can also hold
|
||||
// the preview MOV/MP4, which are not part of the EXR sequence.
|
||||
if (!name.StartsWith(pattern.Prefix, StringComparison.OrdinalIgnoreCase)) continue;
|
||||
|
||||
var frame = pattern.FrameNumberOf(name);
|
||||
if (frame is null) { nonConforming.Add(name); continue; }
|
||||
matched.Add((frame.Value, path));
|
||||
}
|
||||
|
||||
matched.Sort((a, b) => a.Frame.CompareTo(b.Frame));
|
||||
var files = matched.Select(m => m.Path).ToList();
|
||||
frameNumbers = matched.Select(m => m.Frame).ToList();
|
||||
|
||||
// file_count
|
||||
checks.Add(files.Count == expectedCount
|
||||
? ValidationCheck.Pass("file_count", expectedCount.ToString(), files.Count.ToString())
|
||||
: ValidationCheck.Fail("file_count", expectedCount.ToString(), files.Count.ToString(),
|
||||
$"Expected {expectedCount} frames ({manifest.FrameStart}-{manifest.FrameEnd}), found {files.Count} in {outputDirLocal}"));
|
||||
|
||||
// filename_consistency
|
||||
checks.Add(nonConforming.Count == 0
|
||||
? ValidationCheck.Pass("filename_consistency", pattern.Describe(), "all files match")
|
||||
: ValidationCheck.Fail("filename_consistency", pattern.Describe(), $"{nonConforming.Count} nonconforming",
|
||||
"Files not matching the output pattern: " + string.Join(", ", nonConforming.Take(10)),
|
||||
new { files = nonConforming.Take(50) }));
|
||||
|
||||
if (files.Count == 0) return files;
|
||||
|
||||
// missing_frames / duplicate_frames / first_frame / last_frame
|
||||
var seen = new HashSet<int>();
|
||||
var duplicates = new List<int>();
|
||||
foreach (var f in frameNumbers)
|
||||
{
|
||||
if (!seen.Add(f)) duplicates.Add(f);
|
||||
}
|
||||
var missing = new List<int>();
|
||||
for (var f = manifest.FrameStart; f <= manifest.FrameEnd; f++)
|
||||
{
|
||||
if (!seen.Contains(f)) missing.Add(f);
|
||||
}
|
||||
|
||||
checks.Add(missing.Count == 0
|
||||
? ValidationCheck.Pass("missing_frames", "none", "none")
|
||||
: ValidationCheck.Fail("missing_frames", "none", $"{missing.Count} missing",
|
||||
"Missing frames: " + Summarise(missing), new { missing = missing.Take(500) }));
|
||||
|
||||
checks.Add(duplicates.Count == 0
|
||||
? ValidationCheck.Pass("duplicate_frames", "none", "none")
|
||||
: ValidationCheck.Fail("duplicate_frames", "none", $"{duplicates.Count} duplicated",
|
||||
"Duplicate frame numbers (differing padding?): " + Summarise(duplicates),
|
||||
new { duplicates = duplicates.Take(100) }));
|
||||
|
||||
checks.Add(frameNumbers[0] == manifest.FrameStart
|
||||
? ValidationCheck.Pass("first_frame", manifest.FrameStart.ToString(), frameNumbers[0].ToString())
|
||||
: ValidationCheck.Fail("first_frame", manifest.FrameStart.ToString(), frameNumbers[0].ToString(),
|
||||
$"Sequence starts at {frameNumbers[0]}, expected {manifest.FrameStart}"));
|
||||
|
||||
var last = frameNumbers[^1];
|
||||
checks.Add(last == manifest.FrameEnd
|
||||
? ValidationCheck.Pass("last_frame", manifest.FrameEnd.ToString(), last.ToString())
|
||||
: ValidationCheck.Fail("last_frame", manifest.FrameEnd.ToString(), last.ToString(),
|
||||
$"Sequence ends at {last}, expected {manifest.FrameEnd}"));
|
||||
|
||||
// zero_byte — every file, it is only a directory stat
|
||||
var zeroByte = new List<string>();
|
||||
foreach (var path in files)
|
||||
{
|
||||
var len = new FileInfo(path).Length;
|
||||
totalBytes += len;
|
||||
if (len == 0) zeroByte.Add(Path.GetFileName(path));
|
||||
}
|
||||
checks.Add(zeroByte.Count == 0
|
||||
? ValidationCheck.Pass("zero_byte", "none", "none")
|
||||
: ValidationCheck.Fail("zero_byte", "none", $"{zeroByte.Count} empty",
|
||||
"Zero-byte files: " + string.Join(", ", zeroByte.Take(10)), new { files = zeroByte.Take(50) }));
|
||||
|
||||
return files;
|
||||
}
|
||||
|
||||
// ── Header checks (sampled) ──────────────────────────────────────────────
|
||||
|
||||
private static List<string> PickSamples(List<string> files, int sampleEvery)
|
||||
{
|
||||
var picks = new SortedSet<int> { 0, files.Count / 2, files.Count - 1 };
|
||||
for (var i = 0; i < files.Count; i += sampleEvery) picks.Add(i);
|
||||
return picks.Where(i => i >= 0 && i < files.Count).Select(i => files[i]).ToList();
|
||||
}
|
||||
|
||||
private List<(string Path, ExrHeader Header)> ReadHeaders(List<string> sampled, List<ValidationCheck> checks)
|
||||
{
|
||||
var headers = new List<(string, ExrHeader)>();
|
||||
var unreadable = new List<string>();
|
||||
|
||||
foreach (var path in sampled)
|
||||
{
|
||||
try
|
||||
{
|
||||
headers.Add((path, ExrHeader.Read(path)));
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
unreadable.Add($"{Path.GetFileName(path)}: {ex.Message}");
|
||||
_log.LogWarning("Unreadable EXR header: {Path} ({Error})", path, ex.Message);
|
||||
}
|
||||
}
|
||||
|
||||
checks.Add(unreadable.Count == 0
|
||||
? ValidationCheck.Pass("corrupt", "all sampled frames readable", $"{headers.Count} sampled")
|
||||
: ValidationCheck.Fail("corrupt", "all sampled frames readable", $"{unreadable.Count} unreadable",
|
||||
"Unreadable EXR headers: " + string.Join("; ", unreadable.Take(5)), new { files = unreadable.Take(50) }));
|
||||
|
||||
return headers;
|
||||
}
|
||||
|
||||
private static void CheckResolution(RenderManifest m, List<(string Path, ExrHeader H)> headers, List<ValidationCheck> checks)
|
||||
{
|
||||
if (m.Width <= 0 || m.Height <= 0)
|
||||
{
|
||||
checks.Add(ValidationCheck.Skipped("resolution", "manifest carries no expected resolution"));
|
||||
return;
|
||||
}
|
||||
var expected = $"{m.Width}x{m.Height}";
|
||||
var bad = headers.Where(h => h.H.DataWidth != m.Width || h.H.DataHeight != m.Height).ToList();
|
||||
checks.Add(bad.Count == 0
|
||||
? ValidationCheck.Pass("resolution", expected, expected)
|
||||
: ValidationCheck.Fail("resolution", expected, $"{bad[0].H.DataWidth}x{bad[0].H.DataHeight}",
|
||||
$"{bad.Count} of {headers.Count} sampled frames are not {expected} (first: {Path.GetFileName(bad[0].Path)})"));
|
||||
}
|
||||
|
||||
private static void CheckBitDepthAndChannels(RenderManifest m, List<(string Path, ExrHeader H)> headers, List<ValidationCheck> checks)
|
||||
{
|
||||
var expectedDepth = m.Expected?.BitDepth;
|
||||
var actualDepth = headers[0].H.BitDepth ?? "unknown";
|
||||
if (string.IsNullOrWhiteSpace(expectedDepth))
|
||||
{
|
||||
checks.Add(ValidationCheck.Skipped("bit_depth", $"no expected bit depth configured (rendered {actualDepth})"));
|
||||
}
|
||||
else
|
||||
{
|
||||
var mismatched = headers.Where(h => !string.Equals(h.H.BitDepth, expectedDepth, StringComparison.OrdinalIgnoreCase)).ToList();
|
||||
checks.Add(mismatched.Count == 0
|
||||
? ValidationCheck.Pass("bit_depth", expectedDepth, actualDepth)
|
||||
: ValidationCheck.Fail("bit_depth", expectedDepth, mismatched[0].H.BitDepth ?? "unknown",
|
||||
$"{mismatched.Count} of {headers.Count} sampled frames are not {expectedDepth}"));
|
||||
}
|
||||
|
||||
var hasAlpha = headers[0].H.HasAlpha;
|
||||
var channelList = string.Join(",", headers[0].H.Channels.Select(c => c.Name));
|
||||
if (m.Expected?.Alpha is null)
|
||||
{
|
||||
checks.Add(ValidationCheck.Skipped("alpha_channel", $"no expectation configured (channels: {channelList})"));
|
||||
}
|
||||
else
|
||||
{
|
||||
checks.Add(hasAlpha == m.Expected.Alpha.Value
|
||||
? ValidationCheck.Pass("alpha_channel", m.Expected.Alpha.Value ? "alpha present" : "no alpha", channelList)
|
||||
: ValidationCheck.Fail("alpha_channel",
|
||||
m.Expected.Alpha.Value ? "alpha present" : "no alpha",
|
||||
hasAlpha ? "alpha present" : "no alpha",
|
||||
$"Channel list is {channelList}"));
|
||||
}
|
||||
}
|
||||
|
||||
private static void CheckCompression(RenderManifest m, List<(string Path, ExrHeader H)> headers, List<ValidationCheck> checks)
|
||||
{
|
||||
var actual = headers[0].H.CompressionName;
|
||||
if (headers[0].H.CompressionCode is null)
|
||||
{
|
||||
checks.Add(ValidationCheck.Fail("exr_type", "readable EXR", "no compression attribute",
|
||||
"Sampled frame has no compression attribute — not a valid EXR header"));
|
||||
return;
|
||||
}
|
||||
var expected = m.Expected?.ExrCompression;
|
||||
if (string.IsNullOrWhiteSpace(expected))
|
||||
{
|
||||
checks.Add(ValidationCheck.Pass("exr_type", "valid EXR", $"{actual}{(headers[0].H.IsTiled ? ", tiled" : ", scanline")}"));
|
||||
return;
|
||||
}
|
||||
// Compression differing but readable is a WARN, never a stop (§8.1).
|
||||
checks.Add(string.Equals(actual, expected, StringComparison.OrdinalIgnoreCase)
|
||||
? ValidationCheck.Pass("exr_type", expected, actual)
|
||||
: ValidationCheck.Warn("exr_type", expected, actual,
|
||||
$"Compression is {actual}, expected {expected} — readable, so not blocking"));
|
||||
}
|
||||
|
||||
// ── Timecode: the primary check (§8.1a) ──────────────────────────────────
|
||||
|
||||
private void CheckTimecode(
|
||||
RenderManifest m, List<(string Path, ExrHeader H)> headers, List<int> frameNumbers,
|
||||
ServerConfig config, List<ValidationCheck> checks)
|
||||
{
|
||||
var fps = m.Fps > 0 ? m.Fps : 24;
|
||||
var first = headers[0];
|
||||
|
||||
if (first.H.Timecode is null)
|
||||
{
|
||||
// AE's EXR output module writes timeCode; its absence means the
|
||||
// output module changed, so this is a stop rather than a skip.
|
||||
checks.Add(ValidationCheck.Fail("timecode_plate_match", "timecode present", "absent",
|
||||
$"No timeCode attribute in {Path.GetFileName(first.Path)} — the EXR output module no longer writes timecode",
|
||||
new { attributes = first.H.AttributeNames.Take(40) }));
|
||||
return;
|
||||
}
|
||||
|
||||
var renderTc = first.H.Timecode.Value;
|
||||
var renderFirstFrame = frameNumbers.Count > 0 ? frameNumbers[0] : m.FrameStart;
|
||||
|
||||
// timecode_consistency — sampled frames must advance exactly one frame
|
||||
// each. A constant value is the frame-derived-timecode defect.
|
||||
var drift = new List<string>();
|
||||
for (var i = 0; i < headers.Count; i++)
|
||||
{
|
||||
var h = headers[i].H;
|
||||
if (h.Timecode is null) { drift.Add($"{Path.GetFileName(headers[i].Path)}: no timecode"); continue; }
|
||||
var frameNo = SequencePattern.FrameFromName(Path.GetFileName(headers[i].Path)) ?? (renderFirstFrame + i);
|
||||
var expectedTc = Timecode.FromFrames(renderTc.ToFrames(fps) + (frameNo - renderFirstFrame), fps);
|
||||
if (h.Timecode.Value.ToFrames(fps) != expectedTc.ToFrames(fps))
|
||||
{
|
||||
drift.Add($"frame {frameNo}: {h.Timecode.Value} (expected {expectedTc})");
|
||||
}
|
||||
}
|
||||
checks.Add(drift.Count == 0
|
||||
? ValidationCheck.Pass("timecode_consistency", "monotonic, 1 frame per frame", $"{headers.Count} frames sampled")
|
||||
: ValidationCheck.Fail("timecode_consistency", "monotonic, 1 frame per frame", $"{drift.Count} sampled frames off",
|
||||
"Timecode does not advance with the frames: " + string.Join("; ", drift.Take(5)),
|
||||
new { drift = drift.Take(50) }));
|
||||
|
||||
// timecode_fps
|
||||
if (first.H.FramesPerSecond is double headerFps)
|
||||
{
|
||||
checks.Add(Math.Abs(headerFps - fps) < 0.01
|
||||
? ValidationCheck.Pass("timecode_fps", fps.ToString("0.###"), headerFps.ToString("0.###"))
|
||||
: ValidationCheck.Fail("timecode_fps", fps.ToString("0.###"), headerFps.ToString("0.###"),
|
||||
$"EXR framesPerSecond is {headerFps:0.###} but the comp rendered at {fps:0.###}"));
|
||||
}
|
||||
else
|
||||
{
|
||||
checks.Add(ValidationCheck.Skipped("timecode_fps", "no framesPerSecond attribute in the rendered EXRs"));
|
||||
}
|
||||
|
||||
// timecode_plate_match — reference precedence, rank 1 → 3.
|
||||
var reference = ResolveTimecodeReference(m, config);
|
||||
if (reference is null)
|
||||
{
|
||||
checks.Add(ValidationCheck.Warn("timecode_plate_match", "plate timecode", renderTc.ToString(),
|
||||
"No plate reference available: the manifest carries no timing plate, none was found on disk, " +
|
||||
"and no API timecode was supplied. Render timecode is unverified.",
|
||||
new { renderTimecode = renderTc.ToString() }));
|
||||
return;
|
||||
}
|
||||
|
||||
var expectedFrames = reference.PlateTimecode.ToFrames(fps) + (renderFirstFrame - reference.PlateFirstFrame);
|
||||
var expected = Timecode.FromFrames(expectedFrames, fps);
|
||||
var actualFrames = renderTc.ToFrames(fps);
|
||||
var deltaFrames = actualFrames - expectedFrames;
|
||||
var details = new
|
||||
{
|
||||
reference = reference.Kind,
|
||||
platePath = reference.PlatePath,
|
||||
plateTimecode = reference.PlateTimecode.ToString(),
|
||||
plateFirstFrame = reference.PlateFirstFrame,
|
||||
renderFirstFrame,
|
||||
deltaFrames,
|
||||
};
|
||||
|
||||
if (deltaFrames == 0)
|
||||
{
|
||||
// The API fallback proves nothing on its own (§8.1a): it is the
|
||||
// editorial in-point minus a handle count that may not be this
|
||||
// plate's, so a match is reported as a WARN, never a PASS.
|
||||
checks.Add(reference.Kind == "api"
|
||||
? ValidationCheck.Warn("timecode_plate_match", expected.ToString(), renderTc.ToString(),
|
||||
"Matches the API timecode less handles, but the plate itself could not be read — unverified.", details)
|
||||
: ValidationCheck.Pass("timecode_plate_match", expected.ToString(), renderTc.ToString(),
|
||||
$"Matches {reference.Kind} plate {Path.GetFileName(reference.PlatePath ?? "")}", details));
|
||||
return;
|
||||
}
|
||||
|
||||
var message =
|
||||
$"Render starts at {renderTc} but the {reference.Kind} plate says it should be {expected} " +
|
||||
$"({(deltaFrames > 0 ? "+" : "")}{deltaFrames} frames). " +
|
||||
(Math.Abs(deltaFrames) == reference.HandleFrames && reference.HandleFrames > 0
|
||||
? $"That is exactly the {reference.HandleFrames}-frame handle: the comp was anchored to the editorial in-point instead of the plate's first frame."
|
||||
: renderTc.Hours == 0 && renderTc.Minutes == 0
|
||||
? "A near-zero timecode means the comp had no source timecode at all and AE derived it from the frame number."
|
||||
: "");
|
||||
|
||||
checks.Add(reference.Kind == "api"
|
||||
? ValidationCheck.Warn("timecode_plate_match", expected.ToString(), renderTc.ToString(),
|
||||
message + " (checked against the API value, not the plate — treat as advisory)", details)
|
||||
: ValidationCheck.Fail("timecode_plate_match", expected.ToString(), renderTc.ToString(), message, details));
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Reference precedence (§8.1a): manifest timing plate → plate found under a
|
||||
/// configured search root → API timecode minus handles (advisory only).
|
||||
/// </summary>
|
||||
private TimecodeReference? ResolveTimecodeReference(RenderManifest m, ServerConfig config)
|
||||
{
|
||||
var mapper = new PathMapper(_options.PathMappings);
|
||||
var handleFrames = m.Source?.HandleFrames ?? config.ValidationHandleFrames;
|
||||
|
||||
// Rank 1 — the plate the comp was actually built from.
|
||||
var timing = m.Source?.TimingPlate;
|
||||
if (timing is not null && !string.IsNullOrWhiteSpace(timing.Path))
|
||||
{
|
||||
var local = mapper.Map(timing.Path);
|
||||
var header = File.Exists(local) ? ExrHeader.TryRead(local) : null;
|
||||
if (header?.Timecode is not null)
|
||||
{
|
||||
var frame = SequencePattern.FrameFromName(Path.GetFileName(local)) ?? timing.FrameStart ?? m.FrameStart;
|
||||
return new TimecodeReference("manifest", local, header.Timecode.Value, frame, handleFrames);
|
||||
}
|
||||
_log.LogWarning("Timing plate from manifest is unreadable: {Path}", local);
|
||||
}
|
||||
|
||||
// Rank 2 — derive the plate from a configured search root.
|
||||
var derived = FindPlateOnDisk(m, mapper);
|
||||
if (derived is not null) return derived with { HandleFrames = handleFrames };
|
||||
|
||||
// Rank 3 — the API's editorial in-point, less handles. Advisory only.
|
||||
var apiTc = Timecode.Parse(m.Expected?.TimecodeStart);
|
||||
if (apiTc is not null)
|
||||
{
|
||||
var fps = m.Fps > 0 ? m.Fps : 24;
|
||||
var plateStart = Timecode.FromFrames(apiTc.Value.ToFrames(fps) - handleFrames, fps);
|
||||
return new TimecodeReference("api", null, plateStart, m.FrameStart, handleFrames);
|
||||
}
|
||||
|
||||
return null;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Rank 2: look for the shot's plate folders under the configured roots,
|
||||
/// mirroring the panel's two on-disk layouts ({shotCode}/{element}/ and
|
||||
/// {shotCode}_{element}/). Ambiguity is resolved by frame count; still
|
||||
/// ambiguous means no reference rather than a guess.
|
||||
/// </summary>
|
||||
private TimecodeReference? FindPlateOnDisk(RenderManifest m, PathMapper mapper)
|
||||
{
|
||||
if (_options.PlateSearchRoots.Count == 0 || string.IsNullOrWhiteSpace(m.ShotCode)) return null;
|
||||
|
||||
var candidates = new List<(string Dir, int Count)>();
|
||||
foreach (var rootRaw in _options.PlateSearchRoots)
|
||||
{
|
||||
var root = mapper.Map(rootRaw);
|
||||
if (!Directory.Exists(root)) continue;
|
||||
// The show/episode level is unknown here, so scan the root and one
|
||||
// level below it — bounded, and enough for both layouts.
|
||||
var levels = new List<string> { root };
|
||||
try { levels.AddRange(Directory.EnumerateDirectories(root)); } catch { }
|
||||
|
||||
foreach (var level in levels)
|
||||
{
|
||||
try
|
||||
{
|
||||
var grouped = Path.Combine(level, m.ShotCode);
|
||||
if (Directory.Exists(grouped))
|
||||
{
|
||||
foreach (var element in Directory.EnumerateDirectories(grouped))
|
||||
{
|
||||
AddCandidate(candidates, element);
|
||||
}
|
||||
}
|
||||
foreach (var element in Directory.EnumerateDirectories(level, m.ShotCode + "_*"))
|
||||
{
|
||||
AddCandidate(candidates, element);
|
||||
}
|
||||
}
|
||||
catch { /* unreadable branch — keep scanning the rest */ }
|
||||
}
|
||||
}
|
||||
|
||||
if (candidates.Count == 0) return null;
|
||||
|
||||
// Prefer the element whose frame count matches the render: that is the
|
||||
// one the comp's timing came from.
|
||||
var exact = candidates.Where(c => c.Count == m.TotalFrames).ToList();
|
||||
(string Dir, int Count)? chosen =
|
||||
exact.Count == 1 ? exact[0]
|
||||
: exact.Count == 0 && candidates.Count == 1 ? candidates[0]
|
||||
: null;
|
||||
|
||||
if (chosen is null)
|
||||
{
|
||||
_log.LogWarning("Plate reference ambiguous for {Shot}: {Candidates}",
|
||||
m.ShotCode, string.Join(", ", candidates.Select(c => Path.GetFileName(c.Dir))));
|
||||
return null;
|
||||
}
|
||||
|
||||
var firstFile = Directory.EnumerateFiles(chosen.Value.Dir, "*.exr").OrderBy(f => f, StringComparer.Ordinal).FirstOrDefault();
|
||||
if (firstFile is null) return null;
|
||||
var header = ExrHeader.TryRead(firstFile);
|
||||
if (header?.Timecode is null) return null;
|
||||
|
||||
var frame = SequencePattern.FrameFromName(Path.GetFileName(firstFile)) ?? m.FrameStart;
|
||||
return new TimecodeReference("derived", firstFile, header.Timecode.Value, frame, 0);
|
||||
}
|
||||
|
||||
private static void AddCandidate(List<(string Dir, int Count)> candidates, string dir)
|
||||
{
|
||||
try
|
||||
{
|
||||
var count = Directory.EnumerateFiles(dir, "*.exr").Count();
|
||||
if (count > 0) candidates.Add((dir, count));
|
||||
}
|
||||
catch { }
|
||||
}
|
||||
|
||||
// ── Checksums ────────────────────────────────────────────────────────────
|
||||
|
||||
/// <summary>
|
||||
/// Sequence digest = xxHash64 over the per-file digests in frame order
|
||||
/// (§8.2). Reused by the delivery builder to verify what it linked/copied.
|
||||
/// </summary>
|
||||
private string? ComputeSequenceChecksum(List<string> files)
|
||||
{
|
||||
try
|
||||
{
|
||||
var state = new XxHash64.State(0);
|
||||
foreach (var f in files)
|
||||
{
|
||||
var digest = XxHash64.HashFile(f);
|
||||
state.Update(System.Text.Encoding.ASCII.GetBytes(digest));
|
||||
}
|
||||
return state.Digest().ToString("x16");
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_log.LogWarning(ex, "Sequence checksum failed");
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
private static string Summarise(IEnumerable<int> numbers)
|
||||
{
|
||||
var list = numbers.Take(12).ToList();
|
||||
var text = string.Join(", ", list);
|
||||
return numbers.Count() > 12 ? text + ", …" : text;
|
||||
}
|
||||
}
|
||||
|
||||
public sealed record TimecodeReference(
|
||||
string Kind, string? PlatePath, Timecode PlateTimecode, int PlateFirstFrame, int HandleFrames);
|
||||
|
||||
public sealed record ValidationOutcome(
|
||||
List<ValidationCheck> Checks, int FileCount, long TotalBytes, string? Checksum)
|
||||
{
|
||||
public bool HasFailure => Checks.Any(c => c.Status == "FAIL");
|
||||
}
|
||||
|
||||
/// <summary>One row of the E12 report.</summary>
|
||||
public sealed class ValidationCheck
|
||||
{
|
||||
public string CheckName { get; set; } = "";
|
||||
public string Status { get; set; } = "PASS";
|
||||
public string? Expected { get; set; }
|
||||
public string? Actual { get; set; }
|
||||
public string? Message { get; set; }
|
||||
public object? Details { get; set; }
|
||||
|
||||
public static ValidationCheck Pass(string name, string? expected = null, string? actual = null, string? message = null, object? details = null) =>
|
||||
new() { CheckName = name, Status = "PASS", Expected = expected, Actual = actual, Message = message, Details = details };
|
||||
|
||||
public static ValidationCheck Fail(string name, string? expected, string? actual, string message, object? details = null) =>
|
||||
new() { CheckName = name, Status = "FAIL", Expected = expected, Actual = actual, Message = message, Details = details };
|
||||
|
||||
public static ValidationCheck Warn(string name, string? expected, string? actual, string message, object? details = null) =>
|
||||
new() { CheckName = name, Status = "WARN", Expected = expected, Actual = actual, Message = message, Details = details };
|
||||
|
||||
public static ValidationCheck Skipped(string name, string message) =>
|
||||
new() { CheckName = name, Status = "SKIPPED", Message = message };
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Parses `{base}.[#####].exr` output patterns and the filenames they produce.
|
||||
/// </summary>
|
||||
public sealed class SequencePattern
|
||||
{
|
||||
public string Prefix { get; private init; } = "";
|
||||
public string Extension { get; private init; } = ".exr";
|
||||
public int Padding { get; private init; }
|
||||
|
||||
private static readonly Regex FrameInName = new(@"\.(\d{2,9})\.[A-Za-z0-9]+$", RegexOptions.Compiled);
|
||||
|
||||
public static SequencePattern FromOutputPattern(string outputPattern)
|
||||
{
|
||||
var pattern = outputPattern ?? "";
|
||||
var ext = Path.GetExtension(pattern);
|
||||
if (string.IsNullOrEmpty(ext)) ext = ".exr";
|
||||
|
||||
var hashStart = pattern.IndexOf('[');
|
||||
var prefix = hashStart > 0 ? pattern[..hashStart].TrimEnd('.') : Path.GetFileNameWithoutExtension(pattern);
|
||||
var padding = 0;
|
||||
foreach (var ch in pattern)
|
||||
{
|
||||
if (ch == '#') padding++;
|
||||
}
|
||||
|
||||
return new SequencePattern { Prefix = prefix, Extension = ext, Padding = padding };
|
||||
}
|
||||
|
||||
public int? FrameNumberOf(string fileName) => FrameFromName(fileName);
|
||||
|
||||
/// <summary>`UNG_102_019_010_cmp_TT_v005.01001.exr` → 1001.</summary>
|
||||
public static int? FrameFromName(string fileName)
|
||||
{
|
||||
var m = FrameInName.Match(fileName);
|
||||
if (!m.Success) return null;
|
||||
return int.TryParse(m.Groups[1].Value, out var n) ? n : null;
|
||||
}
|
||||
|
||||
public string Describe() => $"{Prefix}.[{new string('#', Math.Max(1, Padding))}]{Extension}";
|
||||
}
|
||||
@@ -21,6 +21,12 @@ public sealed class WorkerOptions
|
||||
public string? FfmpegPath { get; set; }
|
||||
public string? OiiotoolPath { get; set; }
|
||||
public List<PathMapping> PathMappings { get; set; } = new();
|
||||
/// <summary>
|
||||
/// Optional footage roots searched for a shot's plates when a manifest
|
||||
/// carries no timing plate (rank 2 of §8.1a) — e.g. ["V:/_FOOTAGE/UNG"].
|
||||
/// Leave empty and old manifests simply fall through to the API fallback.
|
||||
/// </summary>
|
||||
public List<string> PlateSearchRoots { get; set; } = new();
|
||||
public string WorkerVersion { get; set; } = "1.0.0";
|
||||
public string? AeVersion { get; set; }
|
||||
|
||||
|
||||
@@ -61,9 +61,11 @@ public sealed class WorkerService : BackgroundService
|
||||
: "No interactive After Effects — preview jobs enabled");
|
||||
_lastAeOpen = aeOpen;
|
||||
}
|
||||
// Delivery builds are file work with no AE involvement, so they
|
||||
// are always claimable — someone is usually waiting on the folder.
|
||||
var types = aeOpen
|
||||
? new[] { "AE_RENDER" }
|
||||
: new[] { "AE_RENDER", "PREVIEW_ONLY" };
|
||||
? new[] { "AE_RENDER", "DELIVERY_BUILD" }
|
||||
: new[] { "AE_RENDER", "PREVIEW_ONLY", "DELIVERY_BUILD" };
|
||||
|
||||
var job = await _api.ClaimAsync(_machineId!, types, stoppingToken);
|
||||
if (job is not null)
|
||||
|
||||
@@ -0,0 +1,158 @@
|
||||
namespace VFXReviewWorker;
|
||||
|
||||
/// <summary>
|
||||
/// xxHash64 (§8.2 `checksumAlgo: "xxh64"`). Implemented here rather than taken
|
||||
/// from System.IO.Hashing so the worker stays a dependency-free single-file
|
||||
/// publish for offline workstation installs. Verified against the reference
|
||||
/// test vectors in XxHash64Tests.
|
||||
/// </summary>
|
||||
public static class XxHash64
|
||||
{
|
||||
private const ulong Prime1 = 11400714785074694791UL;
|
||||
private const ulong Prime2 = 14029467366897019727UL;
|
||||
private const ulong Prime3 = 1609587929392839161UL;
|
||||
private const ulong Prime4 = 9650029242287828579UL;
|
||||
private const ulong Prime5 = 2870177450012600261UL;
|
||||
|
||||
public static ulong Hash(ReadOnlySpan<byte> data, ulong seed = 0)
|
||||
{
|
||||
var state = new State(seed);
|
||||
state.Update(data);
|
||||
return state.Digest();
|
||||
}
|
||||
|
||||
public static string HashFile(string path, int bufferSize = 1 << 20)
|
||||
{
|
||||
var state = new State(0);
|
||||
using var fs = new FileStream(path, FileMode.Open, FileAccess.Read, FileShare.ReadWrite, bufferSize, FileOptions.SequentialScan);
|
||||
var buffer = new byte[bufferSize];
|
||||
int read;
|
||||
while ((read = fs.Read(buffer, 0, buffer.Length)) > 0)
|
||||
{
|
||||
state.Update(new ReadOnlySpan<byte>(buffer, 0, read));
|
||||
}
|
||||
return state.Digest().ToString("x16");
|
||||
}
|
||||
|
||||
/// <summary>Streaming state so multi-GB sequences hash without being held in memory.</summary>
|
||||
public struct State
|
||||
{
|
||||
private ulong _v1, _v2, _v3, _v4;
|
||||
private readonly ulong _seed;
|
||||
private ulong _length;
|
||||
private readonly byte[] _buffer;
|
||||
private int _bufferLen;
|
||||
|
||||
public State(ulong seed)
|
||||
{
|
||||
_seed = seed;
|
||||
_v1 = seed + Prime1 + Prime2;
|
||||
_v2 = seed + Prime2;
|
||||
_v3 = seed;
|
||||
_v4 = seed - Prime1;
|
||||
_length = 0;
|
||||
_buffer = new byte[32];
|
||||
_bufferLen = 0;
|
||||
}
|
||||
|
||||
public void Update(ReadOnlySpan<byte> data)
|
||||
{
|
||||
_length += (ulong)data.Length;
|
||||
|
||||
if (_bufferLen > 0)
|
||||
{
|
||||
var need = Math.Min(32 - _bufferLen, data.Length);
|
||||
data[..need].CopyTo(_buffer.AsSpan(_bufferLen));
|
||||
_bufferLen += need;
|
||||
data = data[need..];
|
||||
if (_bufferLen < 32) return;
|
||||
ProcessStripe(_buffer);
|
||||
_bufferLen = 0;
|
||||
}
|
||||
|
||||
while (data.Length >= 32)
|
||||
{
|
||||
ProcessStripe(data[..32]);
|
||||
data = data[32..];
|
||||
}
|
||||
|
||||
if (data.Length > 0)
|
||||
{
|
||||
data.CopyTo(_buffer.AsSpan(0));
|
||||
_bufferLen = data.Length;
|
||||
}
|
||||
}
|
||||
|
||||
private void ProcessStripe(ReadOnlySpan<byte> stripe)
|
||||
{
|
||||
_v1 = Round(_v1, BitConverter.ToUInt64(stripe[..8]));
|
||||
_v2 = Round(_v2, BitConverter.ToUInt64(stripe.Slice(8, 8)));
|
||||
_v3 = Round(_v3, BitConverter.ToUInt64(stripe.Slice(16, 8)));
|
||||
_v4 = Round(_v4, BitConverter.ToUInt64(stripe.Slice(24, 8)));
|
||||
}
|
||||
|
||||
public ulong Digest()
|
||||
{
|
||||
ulong h;
|
||||
if (_length >= 32)
|
||||
{
|
||||
h = RotateLeft(_v1, 1) + RotateLeft(_v2, 7) + RotateLeft(_v3, 12) + RotateLeft(_v4, 18);
|
||||
h = MergeRound(h, _v1);
|
||||
h = MergeRound(h, _v2);
|
||||
h = MergeRound(h, _v3);
|
||||
h = MergeRound(h, _v4);
|
||||
}
|
||||
else
|
||||
{
|
||||
h = _seed + Prime5;
|
||||
}
|
||||
|
||||
h += _length;
|
||||
|
||||
var tail = new ReadOnlySpan<byte>(_buffer, 0, _bufferLen);
|
||||
int i = 0;
|
||||
while (i + 8 <= tail.Length)
|
||||
{
|
||||
h ^= Round(0, BitConverter.ToUInt64(tail.Slice(i, 8)));
|
||||
h = RotateLeft(h, 27) * Prime1 + Prime4;
|
||||
i += 8;
|
||||
}
|
||||
if (i + 4 <= tail.Length)
|
||||
{
|
||||
h ^= BitConverter.ToUInt32(tail.Slice(i, 4)) * Prime1;
|
||||
h = RotateLeft(h, 23) * Prime2 + Prime3;
|
||||
i += 4;
|
||||
}
|
||||
while (i < tail.Length)
|
||||
{
|
||||
h ^= tail[i] * Prime5;
|
||||
h = RotateLeft(h, 11) * Prime1;
|
||||
i++;
|
||||
}
|
||||
|
||||
h ^= h >> 33;
|
||||
h *= Prime2;
|
||||
h ^= h >> 29;
|
||||
h *= Prime3;
|
||||
h ^= h >> 32;
|
||||
return h;
|
||||
}
|
||||
|
||||
private static ulong Round(ulong acc, ulong input)
|
||||
{
|
||||
acc += input * Prime2;
|
||||
acc = RotateLeft(acc, 31);
|
||||
acc *= Prime1;
|
||||
return acc;
|
||||
}
|
||||
|
||||
private static ulong MergeRound(ulong acc, ulong val)
|
||||
{
|
||||
acc ^= Round(0, val);
|
||||
acc = acc * Prime1 + Prime4;
|
||||
return acc;
|
||||
}
|
||||
|
||||
private static ulong RotateLeft(ulong value, int bits) => (value << bits) | (value >> (64 - bits));
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user