# Durability, publication, and recovery This document describes the durability guarantees, their storage assumptions, and how interrupted operations recover. For operator commands, see [backup and restore](/docs/migration-backup-restore.md). ## Guarantees 1. An acknowledged submit is durably committed (redb `Durability::Immediate` plus flushed objects and an independently durable metadata archive) under the storage assumptions below. 2. Accepted revisions never reference unpublished objects: manifests are accepted only when their chunks are present, change sets only when their manifests are present. Offline object collection retains every accepted historical object graph and every active upload intent. 3. A submit is entirely accepted or not accepted: one redb transaction. 4. History is immutable; deleting a current file appends a delete record. 5. Interrupted commands cannot silently discard local work: modified or unexpected files are moved to `.scribe/_state/recovery`, never overwritten. Even displaced files that equal their recorded base are retained, so concurrent editor saves remain reachable. Recovery cleanup is explicit. 6. Detected corruption is reported (`doctor`, `verify`), never rewritten. 7. An interrupted `migrate` is restartable: each metadata upgrade commits its converted records together with a durable cursor, schema 3 is recorded in the repository database before the descriptors are rewritten, and a rerun completes whichever step or marker is missing. Repeating a completed upgrade is a no-op, and startup refuses a root with pending upgrades instead of migrating it. 8. Lost responses cannot duplicate operations: the client persists an OperationId before transmission; the server records every outcome under it. ## Publication sequence | step | action | fault point | |---|---|---| | 1 | receive into quarantine (`objects/staging`, same filesystem); hash while receiving | | | 2 | publish chunk batch: rename loose files, flush each touched shard directory and the loose directory once; append pack and flush it; locator commit | before/after-object-publish | | 3 | publish manifests (same mechanism; chunk presence validated first) | | | 4 | publish change set (manifest presence validated first) | | | 5 | prepare the candidate outside the gate: read repository head and branch head, match the lock policy, validate repository config, stage immutable snapshot pages/history in bounded transactions with undo inventory; candidate = head + 1 | | | 6 | publish the candidate's commit envelope (durable), still outside the gate; it links the identities accepted at the branch head and, for a merge, at the merged head, plus the prepared content and revision roots | before/after-commit-envelope | | 7 | enter per-repository gate; confirm repository head, branch head, and policy revision are unchanged, otherwise release the gate and return to step 5 with a fresh candidate (bounded: the eighth attempt reserves the submit turn before preparation, while keeping preparation outside the publication gate) | | | 8 | one short transaction: recheck operation, heads, lock epoch and chain links; publish prepared roots, commit ledger, operation outcome, rename-lock visibility and upload cleanup marker; advance branch head and repository sequence | before/after-metadata-commit | | 9 | flush the committed outbox into the independent archive, publish durable HEAD, then acknowledge | archive-frame-*/archive-cursor-* | Steps 2 to 4 are normally one `put_objects` batch for small submits (2 file flushes and 1 to 2 directory flushes server-side); large submits use as many batches as the 1024-object / 64 MiB batch limits require. No network and no asset hashing happens inside the gate or the transaction. Lock acquire and release are separate short transactions on the same database and therefore serialize with acceptance; the perf report records their latency. Every submit holds the repository submit-turn mutex from final head confirmation through acceptance. The eighth candidate takes this mutex before preparation, preventing another submit from superseding it while leaving the publication gate free. Snapshot traversal, allocation, encoding, and envelope I/O stay outside the publication gate on every attempt. Future pages/history are staged in bounded transactions; only the final metadata transaction makes them accepted/reachable. Rejected or superseded preparation is removed using its durable undo inventory. The gate covers step 7’s confirmation, step 8’s transaction, and its archive acknowledgement barrier. Archive I/O runs after releasing the redb writer. The envelope flush in step 6 goes through the object store's single pack writer, so it may queue behind an unrelated bulk upload; that wait does not extend the gate hold. A candidate whose captured heads move before acquiring the submit turn is discarded before writing its envelope. Lock/policy races after envelope publication can still leave an unreferenced envelope. Object existence is not acceptance: only the metadata transaction accepts history, `doctor` walks accepted commits `1..=head` and verifies every stored object by digest without treating an unreferenced one as a problem, and `verify` reports per object. The `candidates_superseded` counter includes discarded drafts, and the `attempts` field of `submit.committed` records preparation retries. Branch creation validates its fork point the same way: the first-parent walk of the source branch runs before the gate against one metadata snapshot, and the gate rechecks only the source's activity and head bounds. `crates/scribe-server-cli/tests/kill_publication.rs` aborts the server process at each named publication fault point during a submit, restarts it, resolves the unknown outcome from the client, and checks an independent oracle: exactly one accepted change, every acknowledged change readable, no orphan in history. A crash after the envelope is durable but before the gate leaves exactly the unreferenced envelope described above; the replayed submit publishes a fresh envelope and accepts once. `crates/scribe-server/tests/publication.rs` drives concurrent submitters and head churn between candidate capture and submit-turn acquisition through the in-process adapter. ## Flush sequence per platform | platform | file data | directory entry | note | |---|---|---|---| | Linux | `fsync(2)` via `File::sync_all` | open directory, `fsync` | required for rename durability on ext4/xfs | | macOS | `fcntl(F_FULLFSYNC)` (std's `sync_all`) | same on the directory | asks the drive to flush its cache; latency depends on the storage device | | Windows | `FlushFileBuffers` | handle opened with `FILE_FLAG_BACKUP_SEMANTICS`, `FlushFileBuffers`; only `ERROR_INVALID_FUNCTION` (1) and `ERROR_ACCESS_DENIED` (5) mean "not flushed"; all other errors propagate | Both path-based and rooted directory flushes use this classification. The unsupported case relies on NTFS journaling; Windows storage guarantees still require Windows qualification. | Order per batch: staged file data flushed at stage time -> loose renames -> shard directory flushes (once each) and the loose directory flush -> pack appends -> pack file flush (once per pack) -> locator commit (redb flushes its own file) -> metadata commit -> archive data/HEAD publication. The loose flushes precede the pack writer lock, so a batch with no packed object never takes it; every byte an entry points at is durable before the single locator transaction either way. ## Failure model | failure | guarantee | mechanism | |---|---|---| | process crash at any point | acknowledged submits persist; unacknowledged submits are absent or complete; retry resolves | fault-point kill tests; operation records; pack tail truncation to committed length | | OS crash / power loss | same, assuming the storage honours flushes | flush ordering above; redb two-phase commit; requires separate VM or hardware interruption testing | | partial write | detected: envelope length checks, digest verification on read, pack tail truncation | `verify`, `read_object` | | disk full / I/O error | the operation fails before acknowledgement; nothing partial is accepted | every write, flush, rename, and commit result is checked | | corruption of stored bytes | detected on read and by `doctor` (or `verify --full`); reported per object; never repaired silently | digests | | total storage loss | recovery to the last backup; changes after it are lost | `backup`/`restore`; incarnation change forces clients to reconcile | | external workspace writers | fingerprint mismatch detected; files changed during read/upload are refused; races with uncooperative writers remain (documented) | analyze re-stat, re-hash on upload | ## Assumptions - Storage honours flush requests (no write cache lying about completion). - The data root and its `objects/staging` share one filesystem. - Only one process owns a data root at a time: enforced by an exclusive OS file lock on `/LOCK` (`flock`/`LockFileEx`), taken by `serve` and by every offline maintenance command. A second process fails immediately. `checkpoint` and `integrity-status` take no data-root lock: the first asks the running server to do the work, and the second reads saved reports from the runtime directory. - Diagnostic logs are never used for recovery; recovery reads repository state. ## Client-side recovery - Workspace creation: `.scribe/_state/` is populated first and its descriptor published last, as a flushed temporary renamed into place with the directory flushed. The descriptor is the commit point, so creation either completes or leaves a `.scribe/_state/` without one. That state is reported as an interrupted creation by both `open` and a repeated creation, naming the directory: move it aside, or delete it when it holds no local work, then create the workspace again. It is never completed in place, because its database may hold baselines and opened work belonging to an earlier identity. - Submit: the candidate (change set payload, fingerprints, OperationId) is in `.scribe/_state/state.redb` before the submit request is sent. On an unknown outcome the next `scribe submit` asks `operation_outcome`: accepted -> finalize; rejected -> report; no recorded outcome -> replay the exact same operation ID and intent, including merge ancestry and its repository incarnation. Absence in the same incarnation is not proof that a submit cannot still be accepted. If restore has changed the incarnation and no outcome is recorded, `submit` durably archives the old candidate in `.scribe/_state/recovery/submit-.pending` before clearing the pending row. It retains local opened files and tells the user to sync and resolve or revert old bases. Archive publication is exclusive; retries verify existing bytes and never overwrite another archive. Before immutable object uploads, the client persists a separate upload intent ID and sends it with pins. Uploads cannot accept history. They finish before the complete replayable submit candidate is recorded. Interrupted uploads can be re-analyzed and repeated using content-based presence checks. - Sync: the journal is written before any file changes; each batch marks its entries done after their files are in place; the watermark advances only when every entry is done. A rerun resumes. Files that could not be replaced (Windows sharing violations) are reported and retried on the next run. With [repository mounts](/docs/repository-mounts.md), the parent watermark covers the parent tree alone: it publishes once every parent entry is applied, while an unavailable or unfinished child stays reported and is retried by the next sync. - Delete: `scribe delete` records the opened deletion before removing the file (to `.scribe/_state/recovery`, including files that equal their base). An interruption therefore leaves an opened deletion whose file is still present; repeating the delete completes the removal, and `revert` restores the file from the recorded base. A batch answers per path: a refused or failed path is reported as skipped and the remaining paths are still processed. - Moves: each pending move has a durable ID, journal, and dependencies. Multiple completed local moves can await one submit; an interrupted move must be resumed or aborted before another filesystem move starts. Abort dependent moves first, retaining the latest local bytes. Work in transit remains under `.scribe/_state/staging/move-`, outside both the temporary area (cleared on open) and the recovery area (pruned by the user). If that entry is gone as well as the source and the destination, `move --continue` refuses and `move --abort` retires the intent, saying that nothing was restored: the bytes are already lost, and refusing would only block the workspace. Do not revert individual move endpoints. See [move recovery](/docs/client-usage.md#move-files-and-folders). - Merge resolution: a choice for a rename/collision group is saved atomically before its files are changed. `merge --continue` resumes application; `merge --abort` restores the pre-merge target layout and retains displaced files, including edits made after a recorded choice. Unrelated target changes preserve the plan; a grouped merge whose path dependencies changed must be aborted and replanned. - Offline work: `edit --offline` durably records local intent and a `needs_reconcile` marker, without claiming an exclusive lock. Reconciliation checks revisions and acquires required locks but does not rewrite files or baselines. After interruption, retry it; already acquired locks remain held. Submit still performs authoritative validation. See [offline work](/docs/tutorials/work-offline.md). - Restore: a restored server has a new incarnation. The server refuses new submits carrying an old incarnation, preventing reused change numbers from passing stale base checks. The next sync re-registers the workspace (including authenticated ownership lost with the backup) and performs a full reconcile. Opened files remain deferred, and their baselines retain the old incarnation until explicitly resolved or reverted; advancing the workspace incarnation does not make those old bases eligible for submit. `resolve --update` leaves these paths for an explicit choice without fetching potentially lost base objects. `status`, `scan`, and history queries inspect without repairing work. A partial status report retains local findings and marks unavailable server information explicitly; it is not proof that a workspace can submit. Historical `get` reconstructs and verifies one file in a unique sibling staging directory, flushes the verified file, then publishes exclusively to the requested destination and flushes the directories through the platform contract. Existing destinations are never overwritten. If the final directory flush fails, the command reports that the verified file has already been published; inspect it before retrying with a new destination. Workspace state and cache are unchanged. ## Backups `scribe-server backup` copies descriptors, catalog, `auth-state.redb` when present, per-repository metadata databases, and object stores (locator, loose, packs) with a checksum manifest verified after the copy, while holding the ownership lock (no writer exists). A metadata export alone is not a backup. Place backups on independent storage; the recovery window is the backup interval. `restore` verifies every file against the manifest, assigns new incarnations, and runs structural verification before reporting success. Physical backup and restore flush copied files and their directory entries before reporting success. Backup manifests and rewritten restore descriptors/markers use atomic file publication with a directory flush. Metadata import propagates a failed directory flush instead of reporting a successful activation. Their crash tests cover `maintenance-copy-file-durable`, `maintenance-copy-directory-durable`, and the shared `checkpoint-file-durable`, `checkpoint-renamed`, and `checkpoint-directory-durable` publication windows. ### Auth service data The auth data directory is backed up separately, as a copy of the whole directory taken while `scribe-auth serve` is stopped. The password pepper and the revision witness live outside it and outside its backups. Every applied administrative change and every logout advances the auth revision in the same Immediate transaction and then rewrites the witness atomically, so a copy put back by hand is refused at startup (behind its witness) and by every storage server (behind the revision it recorded in `auth-state.redb`); it would otherwise revive credentials revoked since the backup. The one way back is [`scribe-auth restore`](/docs/reference/cli/scribe-auth.md#restore): it copies the backup into a new, empty directory, starts a new incarnation, moves the revision past both the backup and the witness, disables every earlier credential (keeping server credentials only with `--keep-server-credentials`), writes a recovery administrator credential file before committing, then rewrites the witness and writes the marker last. If it stops before writing the marker, remove the new directory and the recovery credential file and run it again. See [rollback protection](/docs/auth.md#rollback-protection). `scribe-auth migrate` (schema 3 to 4) writes the new administrator credential file before converting records and replaces the marker last; its crash tests cover `auth-migrate-bundle-durable`, `auth-migrate-records-durable`, and `auth-migrate-marker-staged`. Offline administrator recovery ([`scribe-auth admin-credential issue`](/docs/reference/cli/scribe-auth.md#admin-credential-issue)) flushes the credential file before its record commits and is covered at `auth-admin-credential-secret-durable`. Neither command adopts an existing credential file. If `migrate` stops after writing the file but before converting records, or offline issuance stops before its commit, remove the file and run the command again; once `migrate` has committed the records, a rerun only writes the witness and the marker. ## Working-copy publication and interrupted planning Sync rebuilds the plan when it finds an unfinished journal, using bounded changed-path pages in either direction. This includes journals whose planning stopped before the first page or between pages: their target alone is not evidence of a complete plan. Before replacing the journal, it persists the minimum and maximum revisions that may have been partially applied. Repeated interruptions and target changes therefore cannot forget files from an earlier partial update. Initial sync, view changes, and restored incarnations still use paged full reconciliation. The watermark advances only after the rebuilt plan has been applied (opened paths remain recorded as deferred). A repository incarnation change discards the old target and triggers reconciliation against the restored repository. Planning persists the target with its first nonempty page, before applying any file. A failure before that page leaves the previous journal and watermark intact. Completed materialization batches are checkpointed before the next batch; the last batch's completion and watermark advance share one Immediate transaction. This keeps partial progress recoverable without advancing past unapplied files. Deferred updates remain recorded until their files are successfully applied. Sync and revert clear the deferred target in the same transaction as the baseline update; a failed or interrupted revert retains its retry target. An accepted submit clears deferred targets it supersedes; recovery of an older acknowledgement preserves any newer sync target. Every sync derives deferred targets from the requested snapshot in bounded batches rather than replaying a target from a newer revision. Full reconciliation flushes deletion runs page by page; shrinking a large tree never buffers the remaining baselines in a single journal append. Before sync or revert replaces a local entry, or a delete removes it, the entry is moved to a unique name under `.scribe/_state/recovery`, with an index entry beside it (`.origin`: the original path, the reason, the entry's operation id and the time) so every retained file can be mapped back. The index is written before the rename, so a crash between the two leaves an index without content, which `scribe recovery list` names and `prune` clears, never content nobody can attribute. No file copy is needed, but recovery consumes disk space until entries are removed, and Scribe never prunes the area on its own: `scribe recovery list` shows what is there and `scribe recovery prune --older-than ` or `--all` removes retained entries; intent archives (`submit-.pending` and workflow archives) and entries with an unreadable index are always kept and reported. Reports associate each displaced path with its recovery location, summarised to a count when more than ten files were displaced at once. Displaced files are retained even when their fingerprint and content hash match the recorded base. A hash check cannot authorize a later deletion: an editor can save between the check and the rename, or keep writing through an open handle after the rename. Moving every displaced entry into indexed recovery keeps those writes reachable. No working-copy entry is moved into temporary storage for automatic deletion. Updates and view narrowing therefore retain the old selected files until explicit pruning; budget space for that growth. Close editors using retained files and review the entries before pruning. File size and modification time remain status optimizations; neither grants permission to discard local content. Non-forced explicit delete verifies the content id before accepting the deletion. Replacement files are published with an exclusive rename, so a concurrent creation at the destination causes a reported failure, not an overwrite. Original contents stay in recovery if publication fails, and retrying sync rebuilds the journal plan. Publication traverses directory handles without following symlinks. Linux and macOS use relative `openat` and exclusive rename operations; Windows opens directories without delete sharing and rejects reparse points. Existing file/directory transitions apply deletions before additions, preserve unexpected local contents, and defer updates that overlap opened paths. Platform-specific recovery tests must run on the corresponding operating system. Sync and whole-tree export neither promise nor attempt power-loss durability for materialized files. `MaterializeOptions::flush_filesystem` is off by default, so a batch does not flush the whole filesystem on its way out. Nothing downstream trusts materialized bytes to be durable: the server is authoritative, the journal and its watermark live in the workspace database and commit durably on their own, an interrupted sync replays every entry not yet marked done, and file size and modification time are status optimizations that never grant permission to discard local content. On macOS and Windows the platform reports whole-filesystem flushing unsupported and the call does nothing, so enabling the option narrows the window only on Linux. There it runs `syncfs`, which writes back every dirty page of the volume, so each batch stalls behind unrelated writers; macOS deliberately avoids `sync(2)` for the same reason. File reconstruction and recovery renames remain ordered by the sync journal. After an OS crash, `scribe scan --verify` detects missing or changed working-copy bytes; restoration is an explicit operation, and existing local contents are retained in recovery. A process interruption still resumes through the sync journal. This policy does not change the mandatory file, directory, and database flushes for server submits or the durable local state. ## Publication error recovery and repository creation A failed pack append invalidates its writer. Before another batch can publish, the store discards uncommitted buffers and reads the durable locator checkpoint. It truncates the checkpoint pack's uncommitted tail and removes newer packs belonging only to the failed batch. Opening a store applies the same rule, so a process killed between a pack rollover and its locator commit leaves no orphan pack behind: the checkpoint is authoritative, and every pack with a higher id is removed before the checkpoint pack is reopened. Accepted objects remain readable, including when an earlier metadata commit had an uncertain return status. Normal appends retain buffered writes and one pack flush per batch. The locator itself is an index. `scribe-server rebuild-locator` recreates it offline under the data-root lock: every pack entry and loose file is re-hashed and only bytes matching their claimed identity are indexed, so a torn or unflushed body can never be mistaken for the object a later upload would otherwise be deduplicated against. Unverifiable entries are reported, not indexed, and the new database replaces the old one only after it is durable. Concurrent loose-object publishers use distinct, exclusively created staging files even for identical content. Each publication flushes both the shard and its parent; it never relies on another publisher to make a new shard durable. Raw and compressed uploads allocate a fresh random name for each staging file, so concurrent uploads cannot claim the same staging file. Cancelling an upload removes only its own staged files. Repository creation flushes every newly created directory link, the descriptor, the populated repository directory, and its parent before marking the catalog row ready. Required flush failures propagate. Windows retains the platform's explicit directory-journaling contract. Structured uploads validate file-state size/content against manifests, each chunk length, and complete file content identity before publication, outside the gate and metadata transaction. Validation memory is bounded by chunk size. Submit rejects linked ancestors during analysis and upload. Permission updates must target the verified local file even if its pathname changes concurrently. Inspection opens catalog, repository metadata, and locator databases using redb's native read-only mode. A database needing recovery is reported instead of repaired. Normal startup may recover database state; `doctor` and `verify` never do. They report staging leftovers, pack tails, and uncommitted packs without cleaning, truncating, or removing them, and leave repositories outside a selection untouched. ## Metadata database loss and retention Writable metadata reads and successful mutation/retry results pass the archive barrier. A publication failure fences that database handle until restart can archive its surviving committed outbox. It does not turn accepted metadata into a definite rejection. If the database is destroyed before archival completion, that unacknowledged interval may be lost; an independent surviving archive protects acknowledged metadata. Asset-volume recovery still needs verified object backups. The default adjacent archive is not a separate failure domain. Offline consolidation verifies a full logical base before advancing retention floor and pruning covered journal rows/segments. Page reclamation retains every historical root. Stream sequences are persisted independently and never reused. Process-kill tests cover append and rollover, cursor publication, base publication, pruning, relocation, both replay codecs’ batch/final commits, and page reclamation. These process tests do not replace power-loss testing on supported storage. See [operator recovery](/docs/migration-backup-restore.md#independent-metadata-recovery-and-retention).