# Operations For launch preparation, use the [production checklist](/docs/production-deployment.md). For incidents, use [troubleshooting](/docs/troubleshooting.md); for planned credential and certificate changes, use [certificates and credentials](/docs/manual/certificates.md). To set up, grow or upgrade a deployment without `scribe-ctl`, see [run Scribe](/docs/manual.md). ## Layout and ownership ```text /SCRIBE_FORMAT descriptor and versions (required feature auth-ownership-v1 once auth-init ran) /LOCK exclusive OS lock held by serve or any maintenance command /catalog.redb /auth-state.redb workspace and operation ownership, newest auth revision seen (only after auth-init) /repos// metadata.redb, objects/, SCRIBE_FORMAT / disposable diagnostics (daily files, bounded retention) / disposable, private: scribe-server.pid, listen, shutdown-request ``` Authoritative data, logs, and runtime files are separate directories. `init` refuses non-empty directories; `serve` refuses directories without a recognized descriptor; no command uses the current directory implicitly. The data-root, backup, and checkpoint directories Scribe creates are private (0700 on Unix). Every runtime directory (`scribe-server`, `scribe-auth`, `scribe-gateway`, `scribe-ctl`) must not be writable by group or other accounts, because its files stop the service and request backups; startup refuses an existing one that they can write. The auth service has its own data directory with the same one-owner rule. Its password pepper and revision witness file live outside that directory and outside its backups; see [rollback protection](/docs/auth.md#rollback-protection). ## Routine health checks The serving process performs [online object integrity checking](/docs/online-integrity.md) by default, at a bounded rate while foreground work is idle. Use `scribe-server integrity-status --runtime-dir ` for saved coverage and findings. This verifies indexed object bytes; the broader offline checks below remain necessary for metadata and pack-inventory consistency. `scribe-server doctor --data-dir ` (alias `scan`) is the complete read-only check: it takes the ownership lock, so it cannot run beside the server, examines metadata consistency and every pack, and re-hashes every object in parallel. Schedule it before each backup and after any unclean shutdown; `--quick` skips chunk hashing for a faster structural pass. Exit 0 healthy, 1 problems (listed, never repaired), 2 could not run. `scribe-server usage --data-dir ` (alias `df`) reports per-repository counts and sizes, the root total, and the filesystem's free space. It is the first look when a disk fills up and the way to size a backup destination. `doctor` also checks the commit chain, operation records, workspace ownership and lock indexes, retirement records, active workspace counts, live and historical manifests, both collision indexes, `change_paths` against `history`, recorded move predecessors, the completeness of `file_lineage`, and packs end to end. Interrupted-publication leftovers are warnings. `--quick` checks headers and lengths and hashes structured objects while skipping chunk hashing. `verify` is the single-threaded structural alternative; `--full` hashes all objects. Both checkers preserve staging leftovers, pack tails, uncommitted packs, and database files, including repositories outside a `--repo` selection. If a database requires recovery, they report it. Normal server startup performs database recovery; stop it before rerunning the checker. Neither command repairs data. For damaged or missing object bytes, use [`scribe-server repair-objects`](/docs/object-repair.md) with a verified donor backup or offline restored root. Preview is the default; `--apply` preserves current history and retains the original store as evidence. No automatic repair occurs. Index repair is `scribe-server rebuild-locator --data-dir [--repo NAME]`. Run it when `doctor` reports pack entries that are not indexed or a repository's `objects/locator.redb` is lost or unreadable. It takes the ownership lock, re-hashes every pack entry and loose file, indexes only bytes that match their identity, replaces the locator atomically, then verifies the repository structurally. Exit 0 when everything was indexed and verified, 1 when entries were skipped or problems remain (both are listed). `backup` also opens the root for inspection only. It never truncates a pack tail, removes an uncommitted pack, or deletes an interrupted repository creation on the source: it copies the bytes that are there and reports those leftovers, so the copy matches what a `doctor` run of the same moment describes. The restored server performs the ordinary startup repair on its own copy. ## Remote workspaces and locks Use [workspace administration](/docs/workspace-administration.md) to list or retire registrations from a second computer and to force-unlock without a checkout. These client operations run while the server is serving, directly or through a gateway. Owners with read access can list and retire their own registrations; listing or retiring other owners' registrations and force-unlocking require Maintainer or namespace-administrator authority. Without auth, any connected client can perform these operations. Auth-administrator credentials alone do not grant storage access. Doctor, backup/restore, repair, and migration remain server-local operations under the exclusive data-root lock; stop the storage server before running them. ## Unfinished branch creations Branch creation reserves its name durably before copying the source tree, so a name is never taken twice. If the client that owns the operation disappears before the branch activates — a lost `.scribe/_state/`, a discarded workspace — the reservation stays and `scribe branch create` reports that the name is already reserved. Nothing else can complete it, and there is no branch deletion. `scribe-server staging-branches --data-dir ` lists those unfinished snapshots with their pending operation ids, staged row counts, and the change they were pinned to. Confirm with the owner that the operation will never be retried (a client that still holds the intent finishes it with `scribe branch --continue`), then release the name: ```sh scribe-server staging-branches --data-dir scribe-server staging-branches --data-dir --repo game --abandon lighting ``` The command takes the ownership lock, deletes the staged rows in bounded durable transactions, and removes the reservation last, so an interrupted run is simply repeated. Active branches are history: they are never listed and never removed. ## Configuration precedence Flag, then `SCRIBE_LOG` (log filter) and `SCRIBE_SERVER_CONFIG` (config path), then the configuration file, then defaults. Relative paths in the file resolve against the file's directory; relative flag paths against the current directory. Startup logs every resolved non-secret path and limit. The full file is shown in [server-usage.md](/docs/server-usage.md). ## Shutdown SIGINT, SIGTERM, or creating `/shutdown-request` stops admission and closes idle connections. A request file left behind by a killed server is discarded at startup, before the run publishes `/listen`; once that address exists, every request written afterwards is honored, and the file is removed again when the server exits. Queued requests stop; started requests get up to `drain_timeout_secs` for network I/O. The deadline interrupts sockets, then the process waits for started storage jobs before releasing its root lock. A stalled disk can extend shutdown; durable commits are never cancelled. Forced termination leaves recoverable state: acknowledged submits are durable and the next start truncates unreferenced pack tails and removes staging leftovers. ## Security Two deployment modes exist; choose one per data root. **Authenticated (recommended beyond one machine).** Every Scribe endpoint (auth, storage, gateway) serves TLS 1.3 only and accepts only the hybrid post-quantum key exchange `X25519MLKEM768`; a peer that cannot offer it, including a TLS-terminating proxy, fails the handshake. Every caller holds its own device key and proves it with mutual TLS: clients present one-hour access certificates issued by the shared `scribe-auth` service, and no bearer secret crosses a Scribe connection. Every operation is authorized per repository through auth, which remains authoritative: revoking a credential or disabling a user takes effect for new admissions within 30 seconds, whatever the certificate's lifetime. Submit and lock attribution comes from the certificate identity, not from a client-supplied author. Clients pin the server's CA on first use and refuse a changed CA. Setup, roles, certificate lifetimes, and revocation are in [auth.md](/docs/auth.md#identity-device-keys-and-certificates). Keep CA keys offline; give each server its own leaf TLS certificate and its own server credential file. Storage servers bound each source address's unauthenticated connections (`[limits].max_pending_per_source`, evicting the oldest pending connection when full), each user's concurrent connections (`[limits].max_connections_per_principal`), and each user's in-flight request bodies (a quarter of the listener's frame budget). The last two are keyed by the user id, not by a gateway that forwards for the user; see [server configuration](/docs/server-usage.md#configuration-file). Another user's operation id answers exactly like an unknown one, so operation ids reveal nothing across accounts. Each storage server records the newest auth revision it has seen and refuses auth answers from behind it, so restoring the auth data directory from an older copy cannot revive revoked credentials. Restore auth only with [`scribe-auth restore`](/docs/reference/cli/scribe-auth.md#restore), which starts a new auth incarnation that storage servers accept (logging `auth.incarnation_changed`); see [rollback protection](/docs/auth.md#rollback-protection). **Plaintext (loopback or tunnel only).** TCP v2 without auth has no authentication: workspace ids and author labels are self-declared, and any peer that can reach the port can submit, lock, and unlock as anyone. The server refuses non-loopback listen addresses unless `allow_non_loopback = true` and logs `server.plaintext_exposed` when it is. Acceptable only behind an authenticated tunnel, for example `ssh -L 7447:127.0.0.1:7447 vcs-host` on each client with `listen = "tcp://127.0.0.1:7447"` on the server. ## Team lock policy Track `.scribe/config.toml` in the repository. Its accepted `locks.rules` on `main` controls exclusive editing for every branch and can match any file type; feature-branch changes remain proposals until accepted on `main`. In an authenticated deployment, changing the ordered active rules requires a Maintainer; profile-only edits and formatting changes preserving those rules require Writer. Profiles in the same config are selected from each workspace's branch; see [repository configuration](/docs/repository-config.md). Inspect effective rules with `scribe lock rules PATH`, not the workspace copy. Locking a file before it is added, a deletion, or a move destination is normal, so those lock rows are bounded rather than refused: a repository may hold up to 4,096 locks (`[limits].max_untracked_locks`) on paths that do not exist at its head, over and above locks on tracked paths. Acquisition beyond that is `limit_exceeded`; submitting the new file, or releasing a lock nobody needs, restores the room. Online `edit` acquires required locks before opening files. During an outage, users can explicitly record local intent with `edit --offline`, then run `reconcile` when connected. This cannot reserve a file while offline. Locks belong to workspaces and do not expire on disconnect. Successful submissions release locks on submitted paths unless `--keep-locks` is selected; other locks remain held until explicitly released. `status --remote` shows ownership and policy alongside local work, and reports unavailable remote information as unknown. See [lock rules](/docs/client-usage.md#mandatory-lock-rules) and [offline recovery](/docs/tutorials/work-offline.md). ## Capacity planning Size the server for retained unique content, repository metadata, and room for new uploads. History is retained indefinitely; deleting a file from the current tree does not reclaim its stored objects. Recorded operation outcomes are also kept forever by design, so that a retry arriving long after its submission resolves to its recorded result instead of applying a second time: budget under 100 bytes per accepted or rejected operation, roughly 35 MB per year at a thousand changes a day. Nothing prunes them; see [on-disk format](/docs/on-disk-format.md). Compression and chunk reuse can reduce physical storage, depending on the assets. The object store caches at most 64 pack read handles per repository, evicting the least recently used handle before opening another. Active readers retain their own handles until they finish. Size process descriptor limits for these caches across active repositories, in-flight readers, databases, and sockets; historical scans do not retain a handle for every pack ever visited. Each workspace needs space for its selected files, cached objects, temporary reconstruction files, and `.scribe/_state/recovery/`. Recovery retains every displaced file, including unchanged baselines, so allow space for old versions from updates and view narrowing. These files remain until explicit recovery pruning. The client caches objects up to 256 KiB plus manifests. Acceptance is one durable database transaction taken under the repository's publication gate, and it writes roughly seven index rows per change-set entry across current state, the fold index, history, change paths, fold history, lineage, and accepted manifests. Gate hold time therefore grows linearly with the entry count, and every other submit on that repository waits for the whole transaction. The change-set format permits 4,000,000 entries in one submit, which would hold the gate for minutes; `[limits].max_submit_entries` caps what this server accepts and defaults to 100,000 entries, a few seconds of hold on the reference hardware and the same budget merge planning and cherry-pick ancestry already use. A submit above the cap is refused with `LIMIT_EXCEEDED` before the gate is taken: it consumes no change sequence number and is recorded as a rejection like any other refusal, so a retry of the same operation id resolves to the same answer. The refusal names the entry count, the cap, and this configuration key, so the user can add and submit fewer paths at a time while an administrator decides whether to raise it. A first population of a repository from an existing project needs no configuration change: `scribe import` submits the tree as a sequence of changes, each within the cap, reading the cap from `server_info` and halving a batch the server still refuses. Point users at it rather than at this key. Raising the value remains available for a one-off import when nobody else is publishing, and for deliberately measuring acceptance at a given change-set size — see the `million-paths` profile below — but the raised cap applies to every submit the server takes, so lower it afterwards. The cap is read at startup and applies per submit, not per repository. Submit latency includes durable filesystem and database flushes. Lock and workspace transactions share the repository's database writer with accepted changes, so concurrent submits can also increase coordination latency. Measure representative workloads on the intended filesystem and hardware before sizing a deployment. The [testing guide](/docs/testing.md#run-performance-checks) explains how to run and compare performance reports. ## Performance profiles Run profiles through `node tools/scenario.js --profile NAME`: - `ci`: 1544 mixed source and asset files, roughly 100 MiB depending on seed; deterministic correctness gates suitable for shared CI. - `million-paths`: 1,000,000 files of 1 to 4 KiB plus medium binaries and a compressible CDC fixture; opt-in on dedicated hardware. `--quick` uses 100,000 small files. Both runs import every path in one submit, above the default per-submit entry cap, so the harness starts its measurement server with `[limits].max_submit_entries` raised to the profile's own path count — measuring acceptance at that size is the point of the profile, so it deliberately does not route the fixture through `scribe import`, which would measure a sequence of smaller acceptances instead. - `scale`: at least 500 GiB of generated compressible/incompressible data; requires an explicit `--disk-budget-gib` large enough for fixtures, history, backup/restore, and every concurrent working copy. Dedicated hardware only. The Rust helper retains direct library measurements; JS owns the service lifecycle and baseline comparison. Performance profiles use plain TCP; authenticated TLS workflows are covered by the collaboration plans. Baselines are machine-specific. Comparison checks dataset hashes, transport, OS, architecture, run settings, durability, and build profile. The default policy also requires matching hostname and CPU count. Matching metadata alone does not establish comparable system load. Old Rust-harness baselines require explicit replacement after the dataset change. ## Windows notes Windows is the primary client platform. The client sets the read-only attribute in `--readonly` workspaces, retries renames and deletes on sharing violations and transient access-denied results from scanners, opens directories without delete sharing, rejects reparse points, and opens long paths with the `\\?\` prefix whether or not `LongPathsEnabled` is set (other tools on the machine may not handle such paths). Directory-handle flushing on NTFS is best effort and reported as such. Windows-specific tests require a Windows runner; results from another platform do not establish Windows coverage. On Windows, private directories Scribe creates (the configuration root, session and credential directories, key output directories) get a protected access list allowing only the creating account, SYSTEM and Administrators, which files created in them inherit. A private file written elsewhere gets the same list right after it is created. Reading a credential file, device key, pepper or witness refuses one that allows any other account more than reading its attributes, or whose directory lets another account delete its entries; the error names that account's SID. Directories that already exist keep their access lists, so a `--config-dir` or `SCRIBE_CONFIG_DIR` you create yourself must be user-private, as the default `%LOCALAPPDATA%\Scribe` is.