Versions, migration, backup, and restore

The offline commands here take the exclusive data-root lock and refuse to run while scribe-server serve or another maintenance command owns the root. restore and restore-checkpoint take the lock of the new root they create. checkpoint is the exception: it runs beside the live server and asks it to take an online backup.

Scribe is unreleased. The versions below describe the current checkout, not a compatibility promise for earlier development revisions. Startup never performs an implicit migration; use only the explicit conversions described here.

Format versions

Startup inspects the data-root descriptor and refuses unknown kinds, newer versions, or unknown required features. It never migrates silently.

Development changes to moves, file lineage, opened-file reconciliation markers, and grouped merge records extend these unreleased encodings directly. Matching version numbers alone do not guarantee compatibility with an earlier checkout. The conversions below are specific schema upgrades, not a general converter for every development layout. Preserve a verified backup and the matching binaries; finish pending workflows with their original build before upgrading where possible. Do not delete workspace state or recovery files to force an upgrade.

Repository opens also require the exact path collision profile above. Serving, inspection, and migrate all refuse a repository whose profile marker is missing, malformed, or unknown; no conversion rebuilds collision indexes for it. Do not add or change the profile marker by hand.

What to back up

Paths are those of the manual layouts; scribe-ctl where and CLUSTER-INFO.txt list the same files for a cluster scribe-ctl runs.

The web search index and preview files need no backup: the web interface rebuilds them. Neither does a runtime directory.

scribe-ctl upgrade backs up, by itself, each root its migrations change (auth, the stores, the web database) into <state_dir>/backups/<stamp>/ (cluster-state/<name>/backups/ by default), and keeps auth's matching witness in its journal until the upgrade completes. Those copies exist for the upgrade's rollback: they leave out the pepper, the gateway ledger and the web object store, and nothing takes them on a schedule. Back the cluster up as the table says as well.

scribe-server backup --to <empty dir>

Copies every durable file: descriptors, catalog.redb, auth-state.redb when present, and for each repository SCRIBE_FORMAT, metadata.redb, objects/SCRIBE_FORMAT, objects/locator.redb, objects/loose/**, and objects/packs/**. Each copy is flushed, source and copy are hashed, and BACKUP_MANIFEST records per-file BLAKE3 and size, the source incarnation, and each repository's head change and commit id. Staging, LOCK, and runtime files are not copied. The destination must be empty. Run doctor first; a backup of a corrupt root is a corrupt backup.

The source is opened for inspection only, like verify and doctor: a backup never repairs the root it copies. What an interrupted publication left — an uncommitted pack tail, packs created after the last locator checkpoint, staging leftovers — and repositories whose creation was interrupted are reported per repository (findings and warnings with --json, WARNING lines in text) and copied exactly as they are. The manifest records each such repository in an interrupted-<repository id> line. A repository whose database was not closed cleanly (a killed server, a power loss) cannot be inspected without repairing it, so backup records its head as unknown, flags needs_recovery, and copies the files as they are; the restored server recovers them at its first start, and verify after restore confirms the result. Nothing past the checkpoint was ever referenced by history, so the copy is consistent; the restored server truncates the tail and removes those packs during its own first start, as it would after any unclean shutdown.

A symbolic link, junction, or other reparse point is not one of those leftovers: no Scribe operation creates one, and a data root contains no links on any platform. There is no faithful copy of one — following it fabricates content the root never held, skipping it drops a file the run reports as copied — so backup refuses the whole run and names the path instead of recording a finding. Anything that is neither a regular file nor a directory is refused the same way.

The auth service is backed up separately, as described next.

Auth backup and restore

Stop scribe-auth serve and copy its whole data directory. Keep three things out of that copy and back them up separately: the password pepper (see password pepper ownership), the revision witness file named by serve --revision-file, and the credential files of administrators, storage servers and gateways. A backup must not carry its own witness: the witness is what proves a copy is older than the live database.

A copy taken before a revocation still lists that credential as live, so serve refuses a database behind its witness, and every storage server refuses answers from a revision older than the newest it has seen (see rollback protection).

Upgrade rollback: the one exception

A cold copy of the stopped auth data directory may be put back as it is only together with a copy of the revision witness taken at the same moment, and only while nothing has changed at auth since both were copied: no sign-in, credential, grant or revocation. That is the situation of a failed upgrade, where auth was stopped, copied and migrated and never served in between. scribe-ctl upgrade rolls back this way (it keeps the matching witness in its journal until the upgrade completes), and so does an upgrade by hand. Put back after anything changed at auth, the pair would revive what was revoked since, and storage servers that saw the newer revision refuse it anyway.

scribe-auth restore

Outside that exception, scribe-auth restore is the only way back, with the service stopped:

scribe-auth restore --from /backups/auth-2026-09-20 --data-dir /srv/scribe/auth-restored \
  --revision-file /srv/scribe/secrets/auth.revision \
  --admin-credential-out /srv/scribe/secrets/admin-recovery.credential \
  --auth tls://auth.example:7443 --auth-ca /srv/scribe/pki/auth-ca/ca.pem

--from is a copy of a stopped auth data directory written by this schema (restore an older schema with the build that wrote it into a scratch directory, run scribe-auth migrate there, and restore that). --data-dir is a new or empty directory; the backup itself is never modified, so an interrupted restore is rerun into a fresh directory. --revision-file is the current witness if it survived; it may be missing. The restore:

• copies the database and starts a new incarnation;
• sets the revision past both the backup's and the witness's;
• revokes every credential that was live in the backup: every session, automation, administrator, service-account and gateway credential, and, unless --keep-server-credentials is given, every storage server credential;
• writes a recovery administrator credential (label restore-recovery, valid 30 days) to the new file --admin-credential-out, naming --auth and --auth-ca, before committing;
• records a restore audit row with the counts, the revision and the incarnation, then writes the witness and the schema marker.

Accounts, passwords, grants, namespace roles, delegate grants and visibility are kept. Storage servers accept the new incarnation and log auth.incarnation_changed. Serve the restored directory with the same pepper and the rewritten witness, then:

1. With the recovery credential, issue a regular administrator credential and revoke the recovery one (see administrator credentials).
2. Issue new credential files for gateways, service accounts and machines, and for storage servers unless --keep-server-credentials kept theirs; restart each service with its new file.
3. Tell users to sign in again; their sessions were revoked.

Every other change made at auth after the backup was taken is lost: new accounts, grants and password resets, and also revoked grants, namespace roles and delegate grants, disabled accounts and visibility made private again. Review scribe-auth access and audit against your records and make those changes again before users sign in. Credential revocations need no repeating: every credential from before the restore is revoked anyway. Output: restored into DIR; N credentials revoked, M server credentials kept; recovery administrator credential saved to FILE. Users sign in again; issue new machine credentials. With --json the result also names the new incarnation, revision and the recovery admin_credential ID; see JSON output.

scribe-server restore --from <backup> --to <empty dir>

If metadata is intact and only object bytes are damaged, consider verified object repair first. It uses a backup as a donor without rolling accepted history back to the backup's date.

Verifies every backup file against the manifest before copying, copies with per-file verification, assigns a new data-root incarnation and a new incarnation to every repository, writes RESTORE_MARKER, and runs structural verification. A backup directory is untrusted input: a path the manifest names must satisfy the canonical path rule and must be a regular file in the backup, so a link planted in a backup cannot copy its target's bytes into the restored root. Restore refuses and names the path, and copies nothing. Clients whose workspaces remember the old incarnation refuse sync until the restored authority is inspected and explicitly accepted with scribe workspace trust-history --accept-current. Their next sync performs a full verified reconcile. Restoring an older backup changes history: any change accepted after the backup is gone, and clients that had those changes may encounter base mismatches and need to reconcile their local work. Repository ids and therefore auth grants are preserved; a restore is recovery of the same repository, not a clone.

For an unresolved submit, first run scribe submit to resolve or archive the saved intent. If a conflict workflow blocks sync, run scribe resolve --abort: on an incarnation change this durably archives the workflow while preserving local files and opened edits, even if its old target objects no longer exist. Then sync and explicitly resolve stale edits. Interrupted branch operations use scribe branch --continue; unknown old intents are archived and must be selected again after sync. See branch recovery.

Moving a data root to another operating system

A data root is platform-independent. Every stored integer is little-endian except the explicitly ordered big-endian segments inside index keys; identity is content, so location, timestamps, and paths never enter a digest; stored paths are canonical UTF-8 with / separators and no PathBuf or OsString reaches disk; every name inside the root is lowercase hex or a fixed ASCII file name, so no case or illegal-character rule of the receiving filesystem applies; the path collision fold is Scribe's pinned Unicode table rather than the filesystem's own case rules; redb records its page size in its own header instead of adopting the host's; and a data root contains no links on any platform, which is why backup refuses one outright. A root initialized on Windows therefore serves unchanged on Linux or macOS, in either direction.

What does not travel with those bytes:

• A running server. Stop serve and every maintenance command first. The copy must be of a quiesced root; database files copied mid-transaction are torn, which is why backup takes the exclusive data-root lock.
• Credentials and configuration kept outside the root. The password pepper lives outside the auth data root by design and scribe-auth serve --pepper-file needs it; copy it, the revision witness, the TLS material, the administrator, server and gateway credential files, and the server configuration separately. A standalone auth service has its own data directory to copy as well. See password pepper ownership.
• Unix permissions. Scribe applies 0600 and 0700 when it creates files, and Windows ACLs do not translate. A copy arrives with the receiving user's umask, so restrict and re-own the root on the new host. Credential files, the pepper and the revision witness are refused on Unix while other accounts can read them (see credential files).
• Binary equivalence. Startup compares descriptor versions, refuses unknown or newer ones, and never migrates implicitly. Use the same revision on the new host, or run scribe-server migrate there explicitly.

Prefer a plain cold copy when moving one server to a new host: it preserves the data-root incarnation, so clients continue without a full reconcile. Use backup and restore when per-file verification against BACKUP_MANIFEST matters more, accepting the new incarnation and the reconcile it causes. Either way, copy the whole tree with a tool that preserves structure and empty directories (robocopy /E, rsync -a, or a tar round trip), then run doctor --quick and verify --full on the new host before serving. LOCK carries no state: it is a live OS lock that dies with its process, so a copied lock file is inert. Staging leftovers are cleaned at the first start, as after any unclean shutdown.

scribe-server export-metadata --repo <name> --to <file>

Engine-neutral logical dump: "SCMX", version 1, repository id, then every table as (key, value) byte rows, followed by a BLAKE3 trailer over the whole body. Values are the Scribe record encodings (little-endian, versioned), so a future engine or redb major version can import them without reading redb pages. Rows and the running digest are streamed directly to disk. It is not a backup: objects are not included.

scribe-server import-metadata --repo <name> --from <file>

Requires the repository directory to have its object store but no metadata.redb. Builds a new database from the export, renames it into place, and runs structural verification. A tampered export fails the checksum and leaves no installed metadata database. Rows are decoded into batches of at most 1,024 rows or 4 MiB (one larger bounded row may stand alone); keys are limited to 1 MiB and values to 16 MiB. Declared row counts never size an allocation. Failed imports remove their temporary database. Structural verification streams operation rows and history membership, retaining manifest witnesses only for one change set at a time.

scribe-server rebuild-locator [--repo <name>]

Recreates objects/locator.redb of one or every repository from its packs and loose files while the server is stopped. Every pack entry and loose file is re-hashed; only bytes that match the identity they claim are indexed, and the rest are listed as skipped so a later upload of the real object is stored instead of being deduplicated against garbage. The new database is written as locator.rebuild.redb, flushed, and renamed into place, so an interrupted run leaves the previous locator untouched. Each repository is then opened and structurally verified. Exit 0 when nothing was skipped and verification passed, 1 otherwise; nothing else in the store is rewritten.

scribe-server staging-branches [--repo <name>] [--abandon <branch>]

Lists branch creations that reserved a name but never activated, and discards one of them. Listing is read-only. --abandon requires --repo, accepts a branch name, branch id, or pending operation id, and refuses a selector that matches more than one snapshot or an active branch. The staged rows go first, in bounded durable transactions, and the reservation last, so an interrupted run leaves a smaller staging branch and is simply repeated. Objects are not touched: only the staged root reference belongs to the unfinished branch; immutable metadata pages remain shared. See staging lifecycle.

scribe-server migrate

For a cluster scribe-ctl manages, natively or in Docker, scribe-ctl upgrade runs this command with the new release after draining the cluster and backing up each root it changes, and rolls back on failure (upgrade to a new release); under Docker it runs the new storage image as a one-time container with the same data mount. The same sequence by hand is in upgrade a deployment by hand.

scribe-server migrate --data-dir DIR --dry-run                # report only
scribe-server migrate --data-dir DIR                          # apply every required upgrade

The default command plans every catalog repository and runs supported steps in order under one exclusive data-root lock. Run the preview, keep a verified backup with its matching binaries, apply, then run doctor --quick and verify --full before restarting services. Startup refuses pending upgrades. Interrupted runs resume from durable checkpoints; repeating a completed upgrade is a no-op. Unknown versions, incompatible layouts, and unfinished repository creation are refused. A dry run never performs database recovery or writes.

--to current is the default. --to schema-3 stops before registered metadata upgrades. Completed steps are never undone, and a root stopped short of current cannot serve with this build.

This build requires shared snapshot profile 3 and the current Unicode path profile. Repositories from pre-snapshot development builds, including the old schema-0 generation layout, are incompatible: migrate refuses a root or repository at schema 0 and a repository directory that still holds MIGRATION, metadata.redb.gen1, or metadata.redb.gen2. There is no converter or rollback mode for those formats. Keep the data with its matching build; do not add profile markers manually. Missing or malformed snapshot tables are refused, never silently replaced with an empty tree. Asset identities are unchanged.

Within the snapshot profile, explicit migration can complete interrupted schema-1/2 branch-table and schema-marker work. It verifies existing snapshots before publishing schema 3, then rewrites repository and data-root descriptors. Rerunning completes whichever marker is missing; serving and inspection require schema 3 in both descriptor and database. No historical collision index is built.

See the migration framework for adding future upgrades.

Metadata revision 1: workspace-state-v1. Converts the pre-owner workspace record layout and creates owner, owner count, and workspace lock indexes, in pages of 1,024 source rows. Permanent claims in the storage root's auth-state.redb supply existing owners; unclaimed workspaces remain unowned. The old layout is selected by an absent lifecycle table, never by trying both decoders. Each converted record and its indexes commit with the page cursor. Verification completes before recording revision 1. An existing valid workspace_state table is adopted without rebuilding retirement fences or operation outcomes; corruption is refused. Repository ids, branch ids, workspace ids, branch/view/label/timestamps, locks, and accepted history are preserved. Fresh repositories start at the current revision; schema-3 repositories predating this framework must explicitly migrate.

Metadata revision 2: path-fold-v1. Verifies the current Unicode collision profile and shared snapshots before marking the step complete. Missing or incompatible profiles are refused without rebuilding old development indexes. Verification never rewrites paths, accepted history, or objects.

Metadata revision 3: change-notes-v1. Creates the empty change_notes journal that holds descriptions recorded after a change was accepted. Nothing is converted: existing repositories gain the table in one bounded transaction and every commit keeps its message. A journal left behind by an interrupted run is adopted only when every row verifies; a malformed one is refused, since descriptions are user-authored and cannot be rebuilt from other data.

Atomic publication atomic-publication-v1 to v2. Runs before the schema and metadata revisions, once per repository whose descriptor still names v1. A v1 staged rename-lock transfer is exactly a v2 lock acquisition of the same record, and v1 had no staged releases. Bounded transactions move rows from submit_lock_transfers to submit_lock_updates; the final transaction replaces transfer-count/transfer-owner with lock-update-count, lock-release-count (zero), and lock-update-owner, and drops the v1 table. Counters that disagree with their rows are refused before any row moves, and a database mixing both layouts is corruption. The repository descriptor names v2 only after its database converted, and the root descriptor last. Interrupted runs resume from the rows left. Checkpoints taken before the upgrade restore v1 metadata, so take a new checkpoint afterwards.

Assets are never rewritten by a metadata migration; object ids and manifests are unchanged.

Placement

• Keep at least one backup on storage independent of the server's disks.
• The recovery window is the backup interval.
• Rehearse restore on the real project before trusting it, and keep any pre-Scribe authority until import, history, backup, and restore have been rehearsed.

Online incremental checkpoints

The running storage process can back up accepted metadata and content without releasing ownership of its data root:

scribe-server checkpoint --runtime-dir /srv/scribe/run --to /backups/game
scribe-server restore-checkpoint --from /backups/game --to /srv/restored-game

Repeat checkpoint with the same destination for incremental backups. The operator must have access to the server's local runtime directory. A dedicated worker in the owning server handles one checkpoint at a time; requests do not add a network administration endpoint. This works with authenticated and unauthenticated servers. Authenticated storage checkpoints include permanent ownership claims. Back up the central auth service, its external pepper and its revision witness separately as described in auth backup and restore.

Metadata read snapshots are pinned before asset copying. The destination keeps a content-addressed object store per repository, and subsequent checkpoints transfer only missing identities. Object inventories are streamed; unchanged asset payloads are not read from the source. Metadata segments continue the previous committed journal cursor. The first checkpoint exports a self-contained logical base. If offline consolidation has advanced any source floor beyond the preceding checkpoint’s cursor, the next checkpoint starts a fresh full base for every database. Otherwise, it exports only the contiguous journal tail.

A completed checkpoint has an immutable checkpoints/<id>/manifest.json and checksummed .scrj segments. The manifest links to the previous checkpoint's identity and digest. LATEST is published only after metadata and referenced content are durable. Incomplete checkpoints never replace the last complete backup. Retain the chain back to the latest full-base manifest and its shared object stores. Earlier full bases and their tails are separate restore points; deleting them is an explicit backup-retention decision. The backup directory has its own exclusive ownership lock.

Restore requires a destination that does not exist. It replays the checkpoint chain into a temporary data root, verifies copied content and repository structure, assigns new incarnations, and publishes the completed directory. Corruption leaves the destination unpublished and the temporary directory available for diagnosis. Restoring an older checkpoint never accepts orphaned uploaded objects. Operation outcomes remain part of the recovered metadata.

Metadata databases require recovery journal profile 2, snapshot profile 3, commit envelopes v5, and atomic-publication-v2. scribe-server migrate upgrades atomic-publication-v1 stores; other earlier development stores are incompatible with this build and must be kept with their matching binaries. There is no automatic journal initialization or accepted-envelope rewrite. Submission staging and upload pins are authoritative metadata and are included in journal exports and checkpoints. Replay preserves invisible preparations; startup or recover-submissions finishes their bounded cleanup.

scribe-server compact-objects

Object collection is an explicit offline operation. Stop the server and reserve space for a copy of retained content in full mode. For selective payload reclamation use compact-objects --incremental --rewrite-budget-mib 1024. It rewrites only packs below 80% live within that source-byte budget and links unchanged payloads. Budget zero still removes dead locators and wholly dead payloads. Mark/locator indexes need additional space proportional to object count; structured decoding and staging add bounded working space. Hard links require the scratch store on the same filesystem. This is still offline, and a pass scans repository reachability; it is not a time-bounded maintenance window or 50-TB performance qualification.

Rerun the same mode after interruption to resume marking and copying; once the swap marker exists, either mode can complete installation. Serving and locator rebuild refuse a pending swap, so finish it with compact-objects before either. Reused bytes are not scrubbed; use doctor or full compaction when all retained payloads must be verified. Existing checkpoint object stores remain independent of collection in the live data root.

Independent metadata recovery and retention

Each writable catalog, repository, and storage ownership database archives its committed mutations before acknowledgement. The default archive is beside the database; protect against loss of that volume by relocating it to independent storage. This does not copy asset bytes. Keep ordinary object backups too. The central authentication service remains a separate backup responsibility.

Stop the server for these commands; each holds the exclusive data-root lock:

scribe-server recovery-archive --data-dir /srv/scribe --database catalog.redb --verify
scribe-server recovery-archive --data-dir /srv/scribe --database catalog.redb --relocate /recovery/scribe-catalog
scribe-server recovery-archive --data-dir /srv/scribe --database catalog.redb --consolidate
scribe-server recovery-archive --data-dir /srv/scribe --database repos/REPOSITORY_ID/metadata.redb --reclaim-pages

Apply relocation/consolidation separately to each repository’s metadata.redb and, when auth is enabled, auth-state.redb. Consolidation publishes and verifies a logical base, then prunes covered journal rows and older segments. It preserves stream identity and sequence positions, including when no journal rows remain. Run it again after interruption. No online pruning or remote archive-consumer coordination is enabled. Exported checkpoints stay independent and valid.

Page reclamation keeps every accepted historical root and removes only unreachable preparation pages using a disk mark set. It needs space for that scratch index. Reclaimed redb space becomes reusable; the database file need not shrink. Accepted history and original retry outcomes are never expired. Finalized submit upload pins are cleaned automatically and on restart. Unresolved upload/resolution intents stay protected until retirement or explicit compact-objects --abandon-workspace; they have no timeout.

To recover a destroyed metadata file while preserving the damaged evidence:

scribe-server recovery-archive --data-dir /srv/scribe --database catalog.redb --restore-to catalog.recovered.redb

The target must be new. The command verifies the retained archive and restores into quarantine with its own archive. It does not replace the live metadata file or accept orphaned objects. Treat this as a database-level recovery tool: verify repository/catalog consistency and retain a backup before an explicit operator installation. For a complete root with new incarnations and verified objects, use restore or restore-checkpoint. Failed or interrupted replay targets must be discarded and recreated, never resumed or served.

Physical restore creates fresh archives from the verified copied metadata under the destination lock; it does not inherit the original server’s external archive location. Logical imports select a verified replacement archive before publishing their metadata. Old archive copies remain available for diagnosis.

This is a clean development-format cut. Old profile-1 recovery/snapshot stores and v4 envelopes are not converted at startup, and no migration for them is included. Keep old stores with matching binaries, or initialize fresh stores for this build. Upgrade server, gateway, CLI and native shared libraries together. The metadata archive protects acknowledged metadata when it survives database loss; an accepted transaction whose archive acknowledgement never completed can still be lost if its database and outbox are destroyed.

Publication latency and large imports

Archive storage latency affects mutation acknowledgement, but reads continue at the last independently archived snapshot. Invisible preparation shares an archive flush across its local durable batches; final acceptance still requires archive durability. The per-repository submit turn serializes durable preparation and lock acquisition/release. A large import can therefore delay other submissions and lock changes for the duration of preparation. Schedule such imports during a maintenance window appropriate to measured local throughput. Reads remain available. Operation-owned overlapping preparation needs a different allocation and cleanup protocol before that serialization can be removed.

Client trust after restore

A workspace pins verified history per branch. A restored incarnation or a head below that pin is refused. After checking the restored authority, run scribe workspace trust-history --accept-current, then sync. This explicitly replaces the trust anchor and invalidates the old proof binding; it changes no working files or sync watermark. Normal sync then proves and applies the restored state with the existing recovery and opened-file protection rules. Initial trust is established from the connected server; use normal TLS trust verification.