perf(delivery): stop re-reading hard links to verify them
Deploy / deploy (push) Successful in 6m25s

A 25-shot episode took about an hour to package when the linking itself
is instant. All of it was verification: every delivered file was read
back and hashed, and at ~100 MB/s over SMB one 2.8 GB shot costs 30 s.

Re-hashing a hard link cannot disagree with validation. The entry is a
second name for the same data, so it re-reads the exact bytes that were
already hashed — half an hour per episode to prove something that cannot
be false. Verification is now proportional to what was actually done:
copies are read back in full, links carry over the validation-time digest.

Linking has one gap hashing covered incidentally: a source changed after
it passed QC, which a link would carry faithfully. Source timestamps are
now checked against the export's validation time, which costs a stat per
file rather than a full read.

"verifyMode": "always-hash" restores unconditional read-back for a
production that requires it.

Worker 1.3.0.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
twotalesanimation
2026-08-14 09:29:04 +02:00
parent 991abe072c
commit af810258e6
6 changed files with 165 additions and 24 deletions
+19 -6
View File
@@ -114,16 +114,28 @@ never claims to have linked when it copied.
## Verification ## Verification
Every file is read back after packaging and hashed, and the sequence digest is Verification is proportional to what the builder actually did, which is what
compared against the one recorded at validation time. `fsutil hardlink list` does keeps a package fast:
not work over SMB, so link identity is never queried — reading the bytes covers
links and copies through one path. | Mode | How it verifies | Cost for a 25-shot episode |
|---|---|---|
| **Hard-linked** (default) | The entry *is* the source data, so the validation-time digest already describes it. Checks each file exists, and that no source was modified after it passed QC. | seconds |
| **Copied** | Read back in full and hashed; the sequence digest must match the one recorded at validation. | ~30 s per 2.8 GB shot |
Re-reading a hard link cannot disagree with validation — it is the same bytes
under a second name — and at ~100 MB/s over SMB it costs roughly half an hour
per episode to prove something that cannot be false. A copy is genuinely
different: it can be truncated or corrupted, so it is always read back.
Set `"verifyMode": "always-hash"` in the delivery config to read everything back
regardless, for a production that requires it contractually.
An item fails, and the whole package fails with it, when: An item fails, and the whole package fails with it, when:
| Failure | Meaning | | Failure | Meaning |
|---|---| |---|---|
| Checksum mismatch | The delivered bytes differ from what passed QC | | Checksum mismatch (copied items) | The delivered bytes differ from what passed QC |
| A source file modified after validation | The render changed after it passed QC — a link would carry the new bytes faithfully |
| Frame count differs from the export record | The render folder changed after QC — someone re-rendered in place | | Frame count differs from the export record | The render folder changed after QC — someone re-rendered in place |
| Source folder missing | The render is not where the export says it is | | Source folder missing | The render is not where the export says it is |
| No matching files | The folder exists but holds nothing matching the output pattern | | No matching files | The folder exists but holds nothing matching the output pattern |
@@ -186,7 +198,8 @@ key by key, so a project can override just `deliveryRoot`.
"includeMovs": true, "includeMovs": true,
"checksumAlgo": "xxh64", "checksumAlgo": "xxh64",
"reportFormats": ["csv", "json"], "reportFormats": ["csv", "json"],
"copyMode": "auto" "copyMode": "auto",
"verifyMode": "auto"
} }
``` ```
@@ -123,17 +123,80 @@ public class DeliveryStageTests : IDisposable
} }
[Fact] [Fact]
public async Task ChecksumMismatchFailsTheItem() public async Task ChecksumMismatchFailsACopiedItem()
{ {
WriteFrames(3); WriteFrames(3);
var result = await NewStage().RunAsync(new ClaimedJob { Id = "job1" }, var manifest = Manifest(checksum: "deadbeefdeadbeef", copyMode: "always-copy");
Manifest(checksum: "deadbeefdeadbeef"), "machine1", CancellationToken.None, CancellationToken.None); var result = await NewStage().RunAsync(new ClaimedJob { Id = "job1" }, manifest, "machine1",
CancellationToken.None, CancellationToken.None);
Assert.False(result.Ok); Assert.False(result.Ok);
Assert.Equal("FAILED", result.Items[0].Status); Assert.Equal("FAILED", result.Items[0].Status);
Assert.Contains("Checksum mismatch", result.Items[0].ErrorMessage!); Assert.Contains("Checksum mismatch", result.Items[0].ErrorMessage!);
} }
[Fact]
public async Task HardLinkedItemsAreNotReadBack()
{
// A hard link is the same data as the source, so re-hashing it re-reads
// the exact bytes validation already hashed — half an hour per episode
// over SMB to prove something that cannot be false. A deliberately wrong
// recorded checksum is therefore not consulted on the linked path, and
// the validation-time digest carries over unchanged.
WriteFrames(3);
var result = await NewStage().RunAsync(new ClaimedJob { Id = "job1" },
Manifest(checksum: "deadbeefdeadbeef"), "machine1", CancellationToken.None, CancellationToken.None);
Assert.True(result.Ok);
Assert.Equal("VERIFIED", result.Items[0].Status);
Assert.Equal("hardlink", result.Items[0].LinkMode);
Assert.Equal("deadbeefdeadbeef", result.Items[0].Checksum);
}
[Fact]
public async Task AlwaysHashForcesReadBackEvenWhenLinked()
{
WriteFrames(3);
var manifest = Manifest(checksum: "deadbeefdeadbeef");
manifest.VerifyMode = "always-hash";
var result = await NewStage().RunAsync(new ClaimedJob { Id = "job1" }, manifest, "machine1",
CancellationToken.None, CancellationToken.None);
Assert.False(result.Ok);
Assert.Contains("Checksum mismatch", result.Items[0].ErrorMessage!);
}
[Fact]
public async Task SourceTouchedAfterValidationFailsTheLinkedItem()
{
// The one thing linking cannot rule out: the source changing after it
// passed QC. The link would faithfully carry the new bytes.
WriteFrames(3);
var manifest = Manifest();
manifest.Items[0].ValidatedAt = DateTimeOffset.UtcNow.AddHours(-2);
File.SetLastWriteTimeUtc(
Path.Combine(_sourceDir, "SHOT_001_cmp_TT_v001.01002.exr"), DateTime.UtcNow);
var result = await NewStage().RunAsync(new ClaimedJob { Id = "job1" }, manifest, "machine1",
CancellationToken.None, CancellationToken.None);
Assert.False(result.Ok);
Assert.Contains("modified after validation", result.Items[0].ErrorMessage!);
}
[Fact]
public async Task UntouchedSourcesPassTheValidationTimestampCheck()
{
WriteFrames(3);
var manifest = Manifest();
manifest.Items[0].ValidatedAt = DateTimeOffset.UtcNow.AddHours(1);
var result = await NewStage().RunAsync(new ClaimedJob { Id = "job1" }, manifest, "machine1",
CancellationToken.None, CancellationToken.None);
Assert.True(result.Ok);
}
[Fact] [Fact]
public async Task FrameCountDriftSinceQcFailsTheItem() public async Task FrameCountDriftSinceQcFailsTheItem()
{ {
@@ -163,14 +226,15 @@ public class DeliveryStageTests : IDisposable
public async Task ChecksumMatchesTheValidationTimeDigestForIdenticalContent() public async Task ChecksumMatchesTheValidationTimeDigestForIdenticalContent()
{ {
WriteFrames(3); WriteFrames(3);
var first = await NewStage().RunAsync(new ClaimedJob { Id = "job1" }, Manifest(), "machine1", var first = await NewStage().RunAsync(new ClaimedJob { Id = "job1" },
CancellationToken.None, CancellationToken.None); Manifest(copyMode: "always-copy"), "machine1", CancellationToken.None, CancellationToken.None);
// Feeding the recorded digest back in must verify, which is exactly what // Feeding the recorded digest back in must verify, which is exactly what
// the delivery builder does against Export.checksum. // the delivery builder does against Export.checksum on the copy path.
Directory.Delete(_packageDir, recursive: true); Directory.Delete(_packageDir, recursive: true);
var second = await NewStage().RunAsync(new ClaimedJob { Id = "job1" }, var second = await NewStage().RunAsync(new ClaimedJob { Id = "job1" },
Manifest(checksum: first.Items[0].Checksum), "machine1", CancellationToken.None, CancellationToken.None); Manifest(checksum: first.Items[0].Checksum, copyMode: "always-copy"), "machine1",
CancellationToken.None, CancellationToken.None);
Assert.True(second.Ok); Assert.True(second.Ok);
} }
+46 -9
View File
@@ -37,6 +37,7 @@ public sealed class DeliveryStage
var results = new List<DeliveryItemResult>(); var results = new List<DeliveryItemResult>();
var manifestFiles = new List<object>(); var manifestFiles = new List<object>();
var alwaysCopy = string.Equals(manifest.CopyMode, "always-copy", StringComparison.OrdinalIgnoreCase); var alwaysCopy = string.Equals(manifest.CopyMode, "always-copy", StringComparison.OrdinalIgnoreCase);
var alwaysHash = string.Equals(manifest.VerifyMode, "always-hash", StringComparison.OrdinalIgnoreCase);
_log.LogInformation("Building delivery package {Name} ({Count} shots) at {Dir}", _log.LogInformation("Building delivery package {Name} ({Count} shots) at {Dir}",
manifest.PackageName, manifest.Items.Count, packageDir); manifest.PackageName, manifest.Items.Count, packageDir);
@@ -50,7 +51,7 @@ public sealed class DeliveryStage
await _api.ProgressAsync(job.Id, machineId, (double)i / manifest.Items.Count, i, manifest.Items.Count, null, await _api.ProgressAsync(job.Id, machineId, (double)i / manifest.Items.Count, i, manifest.Items.Count, null,
$"Packaging {item.ShotCode} {item.VersionString} ({i + 1}/{manifest.Items.Count})", shutdown); $"Packaging {item.ShotCode} {item.VersionString} ({i + 1}/{manifest.Items.Count})", shutdown);
var result = BuildItem(item, mapper, alwaysCopy, manifestFiles); var result = BuildItem(item, mapper, alwaysCopy, alwaysHash, manifestFiles);
results.Add(result); results.Add(result);
if (result.Status == "FAILED") if (result.Status == "FAILED")
{ {
@@ -77,7 +78,7 @@ public sealed class DeliveryStage
} }
private DeliveryItemResult BuildItem( private DeliveryItemResult BuildItem(
DeliveryManifestItem item, PathMapper mapper, bool alwaysCopy, List<object> manifestFiles) DeliveryManifestItem item, PathMapper mapper, bool alwaysCopy, bool alwaysHash, List<object> manifestFiles)
{ {
var sourceDir = mapper.Map(item.SourceDir); var sourceDir = mapper.Map(item.SourceDir);
var shotDir = mapper.Map(item.ShotDir); var shotDir = mapper.Map(item.ShotDir);
@@ -121,15 +122,51 @@ public sealed class DeliveryStage
totalBytes += new FileInfo(dest).Length; totalBytes += new FileInfo(dest).Length;
} }
// Verification reads the delivered bytes back, which works the same // Verification is proportional to what was actually done.
// whether the entry is a link or a copy — `fsutil hardlink list` is //
// not available over SMB, so link identity cannot be queried. // A hard link is a second name for the same data, so hashing it
var digest = SequenceChecksum(sources.Select(s => Path.Combine(shotDir, Path.GetFileName(s)))); // re-reads the exact bytes validation already hashed — it cannot
if (item.Checksum is { Length: > 0 } && !string.Equals(digest, item.Checksum, StringComparison.OrdinalIgnoreCase)) // disagree, and at ~100 MB/s over SMB it costs half an hour for an
// episode to prove nothing. A copy is different: it can be
// truncated or corrupted, so it is read back in full.
string? digest;
string verifiedBy;
if (usedCopy || alwaysHash)
{ {
return DeliveryItemResult.Failed(item, digest = SequenceChecksum(sources.Select(s => Path.Combine(shotDir, Path.GetFileName(s))));
$"Checksum mismatch after packaging: expected {item.Checksum}, got {digest}"); if (item.Checksum is { Length: > 0 } && !string.Equals(digest, item.Checksum, StringComparison.OrdinalIgnoreCase))
{
return DeliveryItemResult.Failed(item,
$"Checksum mismatch after packaging: expected {item.Checksum}, got {digest}");
}
verifiedBy = "hash";
} }
else
{
// The one thing linking cannot rule out is the source having
// changed since it passed QC — the link would faithfully carry
// the new bytes. Timestamps answer that for the price of a stat.
if (item.ValidatedAt is DateTimeOffset validatedAt)
{
var touched = sources
.Select(s => new FileInfo(s))
.Where(f => f.LastWriteTimeUtc > validatedAt.UtcDateTime.AddSeconds(1))
.ToList();
if (touched.Count > 0)
{
return DeliveryItemResult.Failed(item,
$"{touched.Count} source file(s) were modified after validation " +
$"({touched[0].Name} at {touched[0].LastWriteTimeUtc:u}, validated {validatedAt.UtcDateTime:u}) — " +
"the render changed after it passed QC");
}
}
// Same data as the source, so the validation-time digest describes
// the delivered files exactly.
digest = item.Checksum;
verifiedBy = "link identity";
}
_log.LogDebug("{Shot}: verified by {How}", item.ShotCode, verifiedBy);
string? movDest = null; string? movDest = null;
if (!string.IsNullOrWhiteSpace(item.DeliveryMovPath)) if (!string.IsNullOrWhiteSpace(item.DeliveryMovPath))
+4
View File
@@ -79,6 +79,8 @@ public sealed class DeliveryManifest
public string ReportFile { get; set; } = ""; public string ReportFile { get; set; } = "";
public bool IncludeMovs { get; set; } = true; public bool IncludeMovs { get; set; } = true;
public string CopyMode { get; set; } = "auto"; public string CopyMode { get; set; } = "auto";
/// <summary>"auto" verifies copies by hash and links by identity; "always-hash" reads everything back.</summary>
public string VerifyMode { get; set; } = "auto";
public string ChecksumAlgo { get; set; } = "xxh64"; public string ChecksumAlgo { get; set; } = "xxh64";
public List<string> ReportFormats { get; set; } = new(); public List<string> ReportFormats { get; set; } = new();
public List<DeliveryManifestItem> Items { get; set; } = new(); public List<DeliveryManifestItem> Items { get; set; } = new();
@@ -98,6 +100,8 @@ public sealed class DeliveryManifestItem
public string? Checksum { get; set; } public string? Checksum { get; set; }
public string? DeliveryMovPath { get; set; } public string? DeliveryMovPath { get; set; }
public string ShotDir { get; set; } = ""; public string ShotDir { get; set; } = "";
/// <summary>When this export's validation ran — lets a linked package detect a source changed since QC.</summary>
public DateTimeOffset? ValidatedAt { get; set; }
} }
/// <summary>PREVIEW_ONLY manifest (server-built, RenderPipeline2 §9).</summary> /// <summary>PREVIEW_ONLY manifest (server-built, RenderPipeline2 §9).</summary>
@@ -15,7 +15,7 @@
every change that gets published. publish.ps1 appends the git SHA via every change that gets published. publish.ps1 appends the git SHA via
SourceRevisionId, giving "1.1.0+feb1c7b". SourceRevisionId, giving "1.1.0+feb1c7b".
--> -->
<Version>1.2.1</Version> <Version>1.3.0</Version>
</PropertyGroup> </PropertyGroup>
<ItemGroup> <ItemGroup>
+24 -1
View File
@@ -25,6 +25,13 @@ export interface DeliveryConfig {
checksumAlgo: string; checksumAlgo: string;
reportFormats: string[]; reportFormats: string[];
copyMode: "auto" | "always-copy"; copyMode: "auto" | "always-copy";
/**
* "auto" verifies copies by hash and hard links by identity — a link is the
* same data as the source, so re-reading it cannot disagree with what
* validation already hashed, and at SAN speeds costs ~30 s per shot to prove
* nothing. "always-hash" reads every delivered file back regardless.
*/
verifyMode: "auto" | "always-hash";
} }
export const DELIVERY_CONFIG_DEFAULT: DeliveryConfig = { export const DELIVERY_CONFIG_DEFAULT: DeliveryConfig = {
@@ -39,6 +46,7 @@ export const DELIVERY_CONFIG_DEFAULT: DeliveryConfig = {
checksumAlgo: "xxh64", checksumAlgo: "xxh64",
reportFormats: ["csv", "json"], reportFormats: ["csv", "json"],
copyMode: "auto", copyMode: "auto",
verifyMode: "auto",
}; };
/** /**
@@ -333,6 +341,8 @@ export interface DeliveryManifestItem {
checksum: string | null; checksum: string | null;
deliveryMovPath: string | null; deliveryMovPath: string | null;
shotDir: string; shotDir: string;
/** When validation last ran for this export; null when it never did. */
validatedAt: string | null;
} }
export interface DeliveryManifest { export interface DeliveryManifest {
@@ -345,6 +355,7 @@ export interface DeliveryManifest {
reportFile: string; reportFile: string;
includeMovs: boolean; includeMovs: boolean;
copyMode: string; copyMode: string;
verifyMode: string;
checksumAlgo: string; checksumAlgo: string;
reportFormats: string[]; reportFormats: string[];
items: DeliveryManifestItem[]; items: DeliveryManifestItem[];
@@ -357,7 +368,17 @@ export async function buildDeliveryJobManifest(
const pkg = await tx.deliveryPackage.findUniqueOrThrow({ const pkg = await tx.deliveryPackage.findUniqueOrThrow({
where: { id: packageId }, where: { id: packageId },
include: { include: {
items: { include: { export: true } }, items: {
include: {
export: {
include: {
// Newest validation run: a linked package uses it to detect a
// source file changed after it passed QC.
validations: { orderBy: { createdAt: "desc" }, take: 1, select: { createdAt: true } },
},
},
},
},
project: { select: { code: true, showId: true } }, project: { select: { code: true, showId: true } },
}, },
}); });
@@ -380,6 +401,7 @@ export async function buildDeliveryJobManifest(
reportFile: applyTokens(snapshot.layout.reportFile, baseTokens), reportFile: applyTokens(snapshot.layout.reportFile, baseTokens),
includeMovs: snapshot.includeMovs, includeMovs: snapshot.includeMovs,
copyMode: snapshot.copyMode ?? "auto", copyMode: snapshot.copyMode ?? "auto",
verifyMode: snapshot.verifyMode ?? "auto",
checksumAlgo: snapshot.checksumAlgo, checksumAlgo: snapshot.checksumAlgo,
reportFormats: snapshot.reportFormats, reportFormats: snapshot.reportFormats,
items: pkg.items.map((item) => ({ items: pkg.items.map((item) => ({
@@ -394,6 +416,7 @@ export async function buildDeliveryJobManifest(
fileCount: item.export.exrFileCount, fileCount: item.export.exrFileCount,
checksum: item.export.checksum, checksum: item.export.checksum,
deliveryMovPath: snapshot.includeMovs ? item.export.deliveryMovPath : null, deliveryMovPath: snapshot.includeMovs ? item.export.deliveryMovPath : null,
validatedAt: item.export.validations[0]?.createdAt.toISOString() ?? null,
shotDir: applyTokens(snapshot.layout.shotDir, { shotDir: applyTokens(snapshot.layout.shotDir, {
...baseTokens, ...baseTokens,
shotCode: item.shotCode, shotCode: item.shotCode,