Durability & recovery

Invariants

An acknowledged submit is durably committed (redb Durability::Immediate plus flushed objects) under the storage assumptions below.
Accepted revisions never reference unpublished objects: manifests are accepted only when their chunks are present, change sets only when their manifests are present, and objects are never deleted.
A submit is entirely accepted or not accepted: one redb transaction.
History is immutable; deleting a current file appends a delete record.
Interrupted commands cannot silently discard local work: modified or unexpected files are moved to .scribe/recovery, never overwritten.
Detected corruption is reported (doctor, verify), never rewritten.
A failed migration leaves the previous generation in place (metadata.redb.gen1) and is restartable from its checkpoint.
Lost responses cannot duplicate operations: the client persists an OperationId before transmission; the server records every outcome under it.

Publication sequence

step	action	fault point
1	receive into quarantine (`objects/staging`, same filesystem); hash while receiving
2	publish chunk batch: rename loose files / append pack; flush pack file and each touched shard directory once; locator commit	before/after-object-publish
3	publish manifests (same mechanism; chunk presence validated first)
4	publish change set (manifest presence validated first)
5	enter per-repository gate; candidate = head + 1
6	publish commit envelope (durable)	before/after-commit-envelope
7	one transaction: recheck operation record, bases, case collisions, locks; write current/current_fold/history/change_paths/commits/operations; head++	before-metadata-commit
8	acknowledge	after-metadata-commit

Steps 2 to 4 are normally one put_objects batch for small submits (2 file flushes and 1 to 2 directory flushes server-side); large submits use as many batches as the 1024-object / 64 MiB batch limits require. No network and no asset hashing happens inside the gate or the transaction. Lock acquire and release are separate short transactions on the same database and therefore serialize with acceptance; the perf report records their latency.

crates/scribe-server-cli/tests/kill_publication.rs aborts the server process at each of the six fault points during a submit, restarts it, resolves the unknown outcome from the client, and checks an independent oracle: exactly one accepted change, every acknowledged change readable, no orphan in history.

Flush sequence per platform

platform	file data	directory entry	note
Linux	`fsync(2)` via `File::sync_all`	open directory, `fsync`	required for rename durability on ext4/xfs
macOS	`fcntl(F_FULLFSYNC)` (std's `sync_all`)	same on the directory	asks the drive to flush its cache; this is why fsync costs 5 to 40 ms here
Windows	`FlushFileBuffers`	handle opened with `FILE_FLAG_BACKUP_SEMANTICS`, `FlushFileBuffers`; refusal reported as "not flushed", not fatal	NTFS journals directory updates; the file flush covers data and metadata. Exercised only by the Windows CI runner.

Order per batch: staged file data flushed at stage time -> renames / pack appends -> pack file flush (once per pack) -> shard directory flushes (once each) -> locator commit (redb flushes its own file) -> metadata commit.

Failure model

failure	guarantee	mechanism
process crash at any point	acknowledged commits persist; unacknowledged submits are absent or complete; retry resolves	fault-point kill tests; operation records; pack tail truncation to committed length
OS crash / power loss	same, assuming the storage honours flushes	flush ordering above; redb two-phase commit; not tested on hardware here
partial write	detected: envelope length checks, digest verification on read, pack tail truncation	`verify`, `read_object`
disk full / I/O error	the operation fails before acknowledgement; nothing partial is accepted	every write, flush, rename, and commit result is checked
corruption of stored bytes	detected on read and by `doctor` (or `verify --full`); reported per object; never repaired silently	digests
total storage loss	recovery to the last backup; changes after it are lost	`backup`/`restore`; incarnation change forces clients to reconcile
external workspace writers	fingerprint mismatch detected; files changed during read/upload are refused; races with uncooperative writers remain (documented)	analyze re-stat, re-hash on upload

Assumptions

Storage honours flush requests (no write cache lying about completion).
The data root and its objects/staging share one filesystem.
Only one process owns a data root at a time: enforced by an exclusive OS file lock on <data>/LOCK (flock/LockFileEx), taken by serve and by every maintenance command. A second process fails immediately.
Diagnostic logs are never used for recovery; recovery reads repository state.

Client-side recovery

Submit: the candidate (change set payload, fingerprints, OperationId) is in .scribe/state.redb before any byte is sent. On an unknown outcome the next scribe submit asks operation_outcome: accepted -> finalize; rejected -> report; unknown to the server -> the candidate is dropped and a fresh submit proceeds.
Sync: the journal is written before any file changes; each batch marks its entries done after their files are in place; the watermark advances only when every entry is done. A rerun resumes. Files that could not be replaced (Windows sharing violations) are reported and retried on the next run.
Restore: a restored server has a new incarnation; clients detect it on the next sync and perform a full reconcile instead of trusting incremental state.

Backups

scribe-server backup copies descriptors, catalog, auth-state.redb when present, per-repository metadata databases, and object stores (locator, loose, packs) with a checksum manifest verified after the copy, while holding the ownership lock (no writer exists). A metadata export alone is not a backup. Place backups on independent storage; the recovery window is the backup interval. restore verifies every file against the manifest, assigns new incarnations, and runs structural verification before reporting success.

Working-copy publication and interrupted planning

Sync rebuilds the complete pinned tree plan when it finds an unfinished v1 journal. This includes journals whose planning stopped before the first page or between pages: their target alone is not evidence of a complete plan. The watermark advances only after the rebuilt plan has been applied (opened paths remain recorded as deferred). A repository incarnation change discards the old target and triggers reconciliation against the restored repository. This recovery policy does not alter v1 journal encodings or schema keys.

Before sync or revert replaces a local entry, or a delete removes it, the entry is moved to a unique name under .scribe/recovery. This also retains previously unchanged files: an editor may save through an existing file handle after the rename, and those writes must remain recoverable. No file copy is needed, but recovery consumes disk space until the user explicitly removes entries they no longer need. Scribe does not automatically prune this area. Reports associate each displaced path with its recovery location.

File size and modification time remain status optimizations; neither grants permission to discard local content. Non-forced explicit delete verifies the content id before accepting the deletion. Replacement files are published with an exclusive rename, so a concurrent creation at the destination causes a reported failure, not an overwrite. Original contents stay in recovery if publication fails, and retrying sync rebuilds the journal plan.

Publication traverses directory handles without following symlinks. Linux and macOS use relative openat and exclusive rename operations; Windows opens directories without delete sharing and rejects reparse points. Existing file/directory transitions apply deletions before additions, preserve unexpected local contents, and defer updates that overlap opened paths. Windows behavior requires execution on Windows; macOS tests do not qualify it.

Working-copy publication does not promise power-loss durability for downloaded files or exports. The batch filesystem flush is best-effort: Linux uses syncfs; macOS and Windows report it unsupported. macOS deliberately avoids sync(2), which flushes the entire host and can stall concurrent clients behind unrelated writes. File reconstruction, recovery renames, and journal ordering are unchanged. After an OS crash, scribe scan --verify detects missing or changed working-copy bytes; restoration is an explicit operation, and existing local contents are retained in recovery. A process interruption still resumes through the sync journal. This policy does not change the mandatory file, directory, and database flushes for server commits or the durable local state.