perf(delivery): stop re-reading hard links to verify them
Deploy / deploy (push) Successful in 6m25s
Deploy / deploy (push) Successful in 6m25s
A 25-shot episode took about an hour to package when the linking itself is instant. All of it was verification: every delivered file was read back and hashed, and at ~100 MB/s over SMB one 2.8 GB shot costs 30 s. Re-hashing a hard link cannot disagree with validation. The entry is a second name for the same data, so it re-reads the exact bytes that were already hashed — half an hour per episode to prove something that cannot be false. Verification is now proportional to what was actually done: copies are read back in full, links carry over the validation-time digest. Linking has one gap hashing covered incidentally: a source changed after it passed QC, which a link would carry faithfully. Source timestamps are now checked against the export's validation time, which costs a stat per file rather than a full read. "verifyMode": "always-hash" restores unconditional read-back for a production that requires it. Worker 1.3.0. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
+19
-6
@@ -114,16 +114,28 @@ never claims to have linked when it copied.
|
||||
|
||||
## Verification
|
||||
|
||||
Every file is read back after packaging and hashed, and the sequence digest is
|
||||
compared against the one recorded at validation time. `fsutil hardlink list` does
|
||||
not work over SMB, so link identity is never queried — reading the bytes covers
|
||||
links and copies through one path.
|
||||
Verification is proportional to what the builder actually did, which is what
|
||||
keeps a package fast:
|
||||
|
||||
| Mode | How it verifies | Cost for a 25-shot episode |
|
||||
|---|---|---|
|
||||
| **Hard-linked** (default) | The entry *is* the source data, so the validation-time digest already describes it. Checks each file exists, and that no source was modified after it passed QC. | seconds |
|
||||
| **Copied** | Read back in full and hashed; the sequence digest must match the one recorded at validation. | ~30 s per 2.8 GB shot |
|
||||
|
||||
Re-reading a hard link cannot disagree with validation — it is the same bytes
|
||||
under a second name — and at ~100 MB/s over SMB it costs roughly half an hour
|
||||
per episode to prove something that cannot be false. A copy is genuinely
|
||||
different: it can be truncated or corrupted, so it is always read back.
|
||||
|
||||
Set `"verifyMode": "always-hash"` in the delivery config to read everything back
|
||||
regardless, for a production that requires it contractually.
|
||||
|
||||
An item fails, and the whole package fails with it, when:
|
||||
|
||||
| Failure | Meaning |
|
||||
|---|---|
|
||||
| Checksum mismatch | The delivered bytes differ from what passed QC |
|
||||
| Checksum mismatch (copied items) | The delivered bytes differ from what passed QC |
|
||||
| A source file modified after validation | The render changed after it passed QC — a link would carry the new bytes faithfully |
|
||||
| Frame count differs from the export record | The render folder changed after QC — someone re-rendered in place |
|
||||
| Source folder missing | The render is not where the export says it is |
|
||||
| No matching files | The folder exists but holds nothing matching the output pattern |
|
||||
@@ -186,7 +198,8 @@ key by key, so a project can override just `deliveryRoot`.
|
||||
"includeMovs": true,
|
||||
"checksumAlgo": "xxh64",
|
||||
"reportFormats": ["csv", "json"],
|
||||
"copyMode": "auto"
|
||||
"copyMode": "auto",
|
||||
"verifyMode": "auto"
|
||||
}
|
||||
```
|
||||
|
||||
|
||||
Reference in New Issue
Block a user