Troubleshooting

Find the symptom, check the cause, then retry the affected operation. Examples use the tutorial deployment; adjust paths and endpoints.

Start with doctor

For a cluster scribe-ctl manages, doctor checks what most sections below ask you to check, without changing anything, and names the fix for each problem it finds: services and ports, the gateway's inventory, data-root permissions and formats, certificate and credential expiry, free space, an interrupted command, and for a remote cluster ssh and the remote's scribe-ctl (missing or out of date).

scribe-ctl --config ./scribe-cluster.toml doctor

It exits 0 when everything passes, 1 with warnings and 3 with a failure. See doctor.

Cannot connect

A connection failure names the address and where it came from, for example could not connect to tcp://127.0.0.1:7447 (default; set --server or SCRIBE_SERVER): Connection refused. The source is --server, SCRIBE_SERVER, the workspace registration, or the built-in default. Run with -v to see endpoint selection, trust resolution, and each connection attempt on stderr.

Check the endpoint: --server overrides SCRIBE_SERVER and the workspace's saved address. Authenticated storage requires tls://.
Check that the service is running and its runtime listen file names the expected address. Confirm host/port reachability from the client machine.
When the message adds the server may expect tls:// instead of tcp:// (or the reverse), the two sides may disagree about TLS: an authenticated server answers only TLS, and a plaintext server closes a TLS hello. Use the scheme the server's listen file reports. A tls:// address that was refused outright carries no such hint; a tcp:// failure always does.
Inspect the service log for startup, certificate, or data-root errors.

For a controller-managed deployment:

scribe-ctl --config ./scribe-cluster.toml status
scribe-ctl --config ./scribe-cluster.toml logs storage-a --lines 50

With several clusters, add --cluster NAME. See connection settings.

Ordinary edit requires the server to check mandatory lock rules. For deliberate offline work, use scribe edit --offline PATH, then scribe reconcile after reconnecting. There is no offline lock reservation. See the offline tutorial.

Symptom	Check and next step
Certificate expired or hostname mismatch	Check the machine clock and certificate names/lifetime. Renew the certificate for the actual endpoint (`scribe-ctl renew` does it for a managed cluster).
Handshake fails with `received fatal alert: HandshakeFailure`, or the service logs `peer is incompatible: NoKxGroupsInCommon`	One side does not offer the hybrid post-quantum key exchange `X25519MLKEM768`, which every Scribe endpoint requires. Usually a TLS-terminating proxy or load balancer, or a non-Scribe TLS client, sits in the path. Pass TLS through to the Scribe service, or connect directly.
`PATH is accessible to other accounts (mode 644); run chmod 600 on it`	On Unix, a credential file, session device key, pepper, or revision witness must grant nothing to group or other accounts. Run `chmod 600 PATH` as its owner. Windows performs no such check.
`DIR is accessible to other accounts (mode 755) and holds private keys; restrict it with chmod 700 DIR`	An auth data directory holds both client CAs' private keys, so auth refuses one that group or other accounts can enter or list. Run the `chmod 700 DIR` it names as the directory's owner (for a container, on the host path it is mounted from). `scribe-ctl` plans and `up` report it before starting anything.
`DIR is writable by other accounts (mode 777); restrict it with chmod go-w`	The directory holding a secret, or a service runtime directory or checkpoint store, can be changed by other accounts. Run `chmod go-w DIR`, or move the file into a private directory. A sticky shared directory is accepted for secret files.
`The CA of tls://… changed.` with `trusted:` and `now:` fingerprints	The server's CA changed without a handover the client could follow (or the client missed the handover overlap). Nothing was sent and trust is unchanged. Verify the new fingerprint with the operator, then run the command the message names, `scribe --server URL trust replace`. For a sign-in CA given at login, use `trust replace --auth-ca NEW_FILE`; for a credential file, get a new file. See when a server's CA changes.
`saved trust for tls://… is from an older version of Scribe and cannot be used`	A trust record written before multi-CA trust. Run the command it names, `scribe --server URL trust forget` (no fingerprint needed for such a record), then connect again and approve the server's CA fingerprint.
`unauthenticated`	Log in again; check whether the user is disabled or the session expired/revoked. Disabling a user or resetting their password revokes every session. An explicit `--credential-file` or `SCRIBE_CREDENTIAL_FILE` overrides the saved login; check that credential with `scribe-auth credentials list`.
`auth_unavailable`	Restore storage-to-auth connectivity and check the storage server's credential file (`[auth] credential`): its expiry, and that the server is enabled. Repeated user login will not repair that connection.
`auth_unavailable`: `the auth service is behind a revision this server already saw`	The auth data directory was restored by copying an older backup, which would revive credentials revoked since. The storage server logs `auth.rollback_detected` and refuses every answer until auth is restored properly. Stop auth and run `scribe-auth restore` from the backup; its new incarnation is accepted (`auth.incarnation_changed`). See rollback protection.
`an unfinished login for USER at AUTH is saved`	A login was written but never answered (wrong `--auth` address, auth service down). If the auth service may have accepted it, retry the same `scribe login USER --auth AUTH` first: the client resolves the saved operation and does not create a second session. If the retry cannot succeed — the address was wrong, or you need another username — run `scribe login --discard`, then log in again. Discarding forgets only the local intent; a session the server may have issued for it stays unused until it expires.
`logout is pending; retry scribe logout`	A logout was sent but its answer was lost. Run `scribe logout` again; auth answers success for a credential it already revoked, so the retry completes.
`not logged in at SERVER`	There is no saved session for that server; nothing to log out. Check `--server` and `scribe whoami`.
Logout says the auth service is untrusted	Login with `--auth-ca FILE` saves a copy of that CA in the session directory (`auth-ca.pem`), so a plain `scribe logout` reuses it. If auth's CA has changed since, pass the current CA with `scribe logout --auth-ca FILE`. A trust failure does not leave a pending logout. A session that has already expired is removed without contacting auth.
`scribe-auth serve` refuses to start: `revision witness belongs to another auth incarnation`	The `--revision-file` given is not the one written for this data directory. Use the witness that `init`, `migrate`, or `restore` wrote for it. If the database really was replaced, run `scribe-auth restore`.
`scribe-auth serve` refuses to start: `auth database is at revision R but its witness records W`	The auth database was restored from an older copy, which would revive revoked credentials. Do not replace or edit the witness; run `scribe-auth restore` from the backup, passing the current witness. See auth backup and restore.
`the [auth] table is empty; set namespace, credential, tls_cert and tls_key, or remove the table`	The storage configuration has an `[auth]` table whose keys are all commented out. Set all four keys, or remove the table to serve plaintext. See server configuration.
`auth requires namespace, credential, TLS certificate and TLS key`	Only some `[auth]` settings resolved from the file and `--auth-`/`--tls-` flags. Supply all four.
`[auth].credential PATH is inside the data directory; keep secrets outside it`	Move the credential file (or `tls_key`) out of the data root: backups and host moves copy it.
`forbidden`	The message names the operation, the role it needs, and the repository as far as the client knows them (for example `submit needs the writer role in repository 'game'`). Ask a repository administrator for that grant; see Access denied.

A desktop link that names a tcp:// server or an http:// web address on a network host is refused (a link cannot name a plaintext (tcp://) server on the network; use tls://); plaintext is accepted only for loopback addresses.

Access denied or repository missing

Check grants using an auth administrator credential, not the user's session. The admin helper wraps scribe-auth with those operator credentials:

auth access --user USERNAME
auth grants list --repo studio/REPOSITORY_ID

access shows everything one account may do; replace REPOSITORY_ID with the full ID to see every grant on one repository. Inaccessible repositories are hidden from discovery. Reader allows sync; writer allows submit; maintainer allows forced unlock; a namespace reader grant reads every repository in the namespace and its inventory; namespace administration allows repository creation. Auth administration alone grants no repository access. See roles.

Submit, sync, or locks block progress

Symptom	Safe next step
Submit outcome unknown, exit 3	Rerun `scribe submit` in the same workspace. Keep its pending state; the client resolves the original operation.
`base_mismatch`	For ordinary stale work, run `scribe resolve --update`, resolve conflicts, then submit. Finish or abort pending moves first. A grouped merge must be aborted and restarted against the new target; retain and reapply recovered edits.
`locked_by_other` / `lock_held`	Inspect `scribe status --remote` or `scribe lock list`; coordinate with the holder. Forced unlock is an explicit maintainer action.
`lock_required`	Inspect `scribe lock rules PATH`; `scribe reconcile` checks opened files and acquires required locks. Accepted `main` rules apply even on another branch.
Sync reports deferred opened paths	Submit the opened work, or revert only when you intend to discard it, then retry sync. Closing an editor does not clear Scribe's opened state.
File replacement fails with a sharing violation	Close applications holding the file and retry; keep the journal and recovery files.
Interrupted move or merge	Resume with `scribe move --continue` or `scribe merge --continue`. Use the matching abort command only when abandoning that work.
Cannot abort a pending move	Inspect move IDs and dependencies in status. Abort later dependent moves first; independent moves can be undone in any order.
Choosing one conflict also selects other paths	Rename/collision groups resolve atomically. Review all members; agreed names select contents at that destination, competing names select the complete layout.
`merge_incomplete`	The submission did not cover every path the source changed since the merge base. Abort the merge and plan it again (`scribe merge --abort`, then `scribe merge <branch>`); resolve each listed path, using `scribe resolve --ours PATH` when you mean to keep the target's version.
Ambiguous file origins or merge planning limit	Do not guess a rename. Abort and review an earlier explicit `--base` for ambiguous lineage; split an oversized merge by source revision. See the documented bounds.
`branch 'NAME' already exists (names are compared ignoring case and Unicode form)`	An active branch holds a colliding name: `Art` and `art` are the same branch name. Choose another name, or use the existing branch.
`scribe branch create` says the name is reserved by an unfinished branch creation	A client still holding the intent finishes it with `scribe branch --continue`. If that client is gone, an operator stops the server and releases the name with `scribe-server staging-branches --abandon`.
`add`, `edit`, or `delete` says to finish conflict resolution	The merge in progress would submit that work in its integration commit. Finish it with `scribe merge --continue`, or discard it with `scribe merge --abort`, then open the path.
`.scribe/_state` has no `SCRIBE_FORMAT`: an interrupted workspace creation	Workspace creation stopped before its descriptor was published. Move that `.scribe/_state` directory aside (delete it only when it holds no local work), then run `scribe workspace create` again.
Old computer's workspace still appears on the server	Run `scribe workspace list --all-repos` from the new computer, then `scribe workspace remove WORKSPACE_ID --repo REPO`. This retires the registration and preserves the old computer's files. See workspace administration.
Workspace removal reports held locks	Inspect `scribe lock list --repo REPO`. A Maintainer can release the intended paths with `scribe lock release --repo REPO --force PATH`, then repeat removal. Without auth, any connected client can do this.
`too many new workspaces this hour; reuse or remove existing workspaces and retry later`	An authenticated server registers at most 64 new workspaces per user per hour (`limit_exceeded`). Reuse an existing working copy, or remove unused registrations with `scribe workspace remove`, and retry later.
`workspace_retired`	The workspace ID was permanently retired. Preserve its files and recovery copies; create a new working copy in a separate directory and copy over the needed local work. Re-registering the old ID cannot undo retirement.
`workspace list --all-repos --everyone` is forbidden	With auth, Maintainer permission is required in every selected repository. Use `--repo REPO` for a repository you administer, or omit `--everyone` to list your own registrations. Auth administration alone does not grant storage access.
Workspace removal lost its connection	Repeat the same command with the same per-user configuration directory. The client resolves the persisted operation outcome before sending a new request.
A selected repository mount is `unavailable`	The parent's own files still sync and its watermark advances. Grant access or exclude the mount, then rerun `scribe sync` to retry the child.

Detailed procedures: client workflows, conflict resolution.

Status is incomplete or work is still pending

Default status checks opened/workflow paths locally. Use --scan for unopened changes and --verify for content hashing. Summary counts overlap because one path can be modified, moved, and need reconciliation at once.

--remote adds current server observations. If a query fails, the CLI keeps the local report, marks remote state unavailable, and exits 1; lock fields remain unknown rather than showing a partially fetched lock list. Local inspection or ignore-rule failures also exit 1 with available findings. Changes, conflicts, and pending work alone do not make status fail.

A resolved merge still needs submission. A saved but unapplied choice needs merge --continue. needs_reconcile records unchecked local intent; reconcile checks it against the server without replacing local bytes. See status fields and workflow hints.

Find or download an older file

Ordinary per-file history follows recorded moves, forks, and merged ancestry. Use scribe log PATH --all-branches to include related revisions on unmerged branches. Keep the returned snapshot and pagination cursor together. Reusing a deleted filename creates a separate file incarnation, not an automatic link to the former file. See history.

For scribe get PATH --revision N --to FILE, use the path and branch that held the file at that snapshot; a current name may not exist before its rename. The destination must be new. The command requires the server and does not change workspace state. If it reports a flush failure after publication, the verified destination may already exist; inspect it before retrying. See historical downloads.

Data root locked, full, or unhealthy

Locked: stop its owner gracefully and wait. Do not delete LOCK; ownership is an OS lock, not the presence of the file.
Disk full: stop the service, inspect scribe-server usage --data-dir DIR, and provide capacity. Do not delete stored objects or database files.
Integrity failure: preserve the root and run offline doctor. Recover from a verified copy through the maintenance procedure.
Locator lost or pack entries not indexed: doctor names the fix; run scribe-server rebuild-locator --data-dir DIR with the server stopped. It re-hashes every stored object and indexes only bytes that verify; skipped entries are listed and exit 1 means the repository still needs attention.
Database recovery required: normal startup performs database recovery. Stop it before rerunning the read-only check. doctor never repairs data.

See offline health checks.

Gateway cannot place or serve a repository

scribe-gateway --config ./scribe-gateway.toml status --json

Check backend health, observation freshness, free space, and the gateway credential's expiry, namespace reader grant, and delegate grant (auth access --user scribe-gateway). Restart the gateway after changing configured membership. An instance that stays unavailable with an authorization error right after a configuration change usually names the wrong namespace: the gateway's service account must hold the namespace reader and delegate grants for the namespace that instance was auth-inited with, and scribe-ctl refuses a gateway entry whose namespace differs from the managed store it points at. A client error saying names are <namespace>/<name> means the gateway serves several namespaces and has no default_namespace; write the qualified name or configure a default.

An unavailable backend leaves its repositories unavailable; the gateway is not failover storage. See placement and reconciliation.

A remote cluster's scribe-ctl is missing or out of date

A remote command runs a scribe-ctl installed in the cluster's directory there. When the remote has this machine's OS and architecture, any command run with --remote installs this machine's own build first, so there is nothing to fix. From a machine of another platform (a macOS laptop and a Linux server, say) it cannot: the remote runs the scribe-ctl a --source command or bootstrap last built there, and one built from an older checkout does not know newer commands (doctor, for one: remote.doctor fails). doctor's remote.ctl names the fix. Refresh it without touching the cluster, its services or a command running there:

scribe-ctl --config ./scribe-cluster.toml bootstrap --source <checkout> --plan
scribe-ctl --config ./scribe-cluster.toml bootstrap --source <checkout>

--source builds scribe-ctl there in Docker from a checkout of this scribe-ctl's version; without docker on the remote, pass --binaries DIR with this release built for the remote. See bootstrap.

If you need help

Record the command, exit code, build version, endpoint, time, and relevant error lines. Include whether the operation is pending and which roots are affected. Share a minimal, reviewed excerpt; exclude credentials and private file contents.