# Troubleshooting

Find the symptom, check the cause, then retry the affected operation. Examples use
the [tutorial deployment](/docs/tutorials.md); adjust paths and endpoints.

> [!WARNING]
> Keep `.scribe/_state/`, recovery files, pending requests, and server data intact while investigating. Deleting state does not resolve an unknown outcome.

## Start with doctor

For a cluster scribe-ctl manages, `doctor` checks what most sections below
ask you to check, without changing anything, and names the fix for each
problem it finds: services and ports, the gateway's inventory, data-root
permissions and formats, certificate and credential expiry, free space, an
interrupted command, and for a remote cluster ssh and the remote's
scribe-ctl ([missing or out of date](/docs/troubleshooting.md#a-remote-clusters-scribe-ctl-is-missing-or-out-of-date)).

```sh
scribe-ctl --config ./scribe-cluster.toml doctor
```

It exits 0 when everything passes, 1 with warnings and 3 with a failure. See
[doctor](/docs/reference/cli/scribe-ctl.md#doctor).

## Cannot connect

A connection failure names the address and where it came from, for example
`could not connect to tcp://127.0.0.1:7447 (default; set --server or SCRIBE_SERVER): Connection refused`.
The source is `--server`, `SCRIBE_SERVER`, the workspace registration, or the
built-in default. Run with `-v` to see endpoint selection, trust resolution,
and each connection attempt on stderr.

1. Check the endpoint: `--server` overrides `SCRIBE_SERVER` and the workspace's
   saved address. Authenticated storage requires `tls://`.
2. Check that the service is running and its runtime `listen` file names the
   expected address. Confirm host/port reachability from the client machine.
3. When the message adds `the server may expect tls:// instead of tcp://` (or
   the reverse), the two sides may disagree about TLS: an authenticated server
   answers only TLS, and a plaintext server closes a TLS hello. Use the scheme
   the server's `listen` file reports. A `tls://` address that was refused
   outright carries no such hint; a `tcp://` failure always does.
4. Inspect the service log for startup, certificate, or data-root errors.

For a controller-managed deployment:

```sh
scribe-ctl --config ./scribe-cluster.toml status
scribe-ctl --config ./scribe-cluster.toml logs storage-a --lines 50
```

With several clusters, add `--cluster NAME`. See [connection settings](/docs/reference/cli/scribe.md#defaults-and-configuration).

Ordinary `edit` requires the server to check mandatory lock rules. For deliberate
offline work, use `scribe edit --offline PATH`, then `scribe reconcile` after
reconnecting. There is no offline lock reservation. See the
[offline tutorial](/docs/tutorials/work-offline.md).

## TLS or login fails

| Symptom | Check and next step |
| --- | --- |
| Certificate expired or hostname mismatch | Check the machine clock and certificate names/lifetime. [Renew the certificate](/docs/manual/certificates.md#server-tls-certificates) for the actual endpoint (`scribe-ctl renew` does it for a managed cluster). |
| Handshake fails with `received fatal alert: HandshakeFailure`, or the service logs `peer is incompatible: NoKxGroupsInCommon` | One side does not offer the hybrid post-quantum key exchange `X25519MLKEM768`, which every Scribe endpoint requires. Usually a TLS-terminating proxy or load balancer, or a non-Scribe TLS client, sits in the path. Pass TLS through to the Scribe service, or connect directly. |
| `PATH is accessible to other accounts (mode 644); run chmod 600 on it` | On Unix, a credential file, session device key, pepper, or revision witness must grant nothing to group or other accounts. Run `chmod 600 PATH` as its owner. Windows performs no such check. |
| `DIR is accessible to other accounts (mode 755) and holds private keys; restrict it with chmod 700 DIR` | An auth data directory holds both client CAs' private keys, so auth refuses one that group or other accounts can enter or list. Run the `chmod 700 DIR` it names as the directory's owner (for a container, on the host path it is mounted from). `scribe-ctl` plans and `up` report it before starting anything. |
| `DIR is writable by other accounts (mode 777); restrict it with chmod go-w` | The directory holding a secret, or a service runtime directory or checkpoint store, can be changed by other accounts. Run `chmod go-w DIR`, or move the file into a private directory. A sticky shared directory is accepted for secret files. |
| `The CA of tls://… changed.` with `trusted:` and `now:` fingerprints | The server's CA changed without a handover the client could follow (or the client missed the handover overlap). Nothing was sent and trust is unchanged. Verify the new fingerprint with the operator, then run the command the message names, `scribe --server URL trust replace`. For a sign-in CA given at login, use `trust replace --auth-ca NEW_FILE`; for a credential file, get a new file. See [when a server's CA changes](/docs/client-usage.md#when-a-servers-ca-changes). |
| `saved trust for tls://… is from an older version of Scribe and cannot be used` | A trust record written before multi-CA trust. Run the command it names, `scribe --server URL trust forget` (no fingerprint needed for such a record), then connect again and approve the server's CA fingerprint. |
| `unauthenticated` | Log in again; check whether the user is disabled or the session expired/revoked. Disabling a user or resetting their password revokes every session. An explicit `--credential-file` or `SCRIBE_CREDENTIAL_FILE` overrides the saved login; check that credential with `scribe-auth credentials list`. |
| `auth_unavailable` | Restore storage-to-auth connectivity and check the storage server's credential file (`[auth] credential`): its expiry, and that the server is enabled. Repeated user login will not repair that connection. |
| `auth_unavailable`: `the auth service is behind a revision this server already saw` | The auth data directory was restored by copying an older backup, which would revive credentials revoked since. The storage server logs `auth.rollback_detected` and refuses every answer until auth is restored properly. Stop auth and run [`scribe-auth restore`](/docs/reference/cli/scribe-auth.md#restore) from the backup; its new incarnation is accepted (`auth.incarnation_changed`). See [rollback protection](/docs/auth.md#rollback-protection). |
| `an unfinished login for USER at AUTH is saved` | A login was written but never answered (wrong `--auth` address, auth service down). If the auth service may have accepted it, retry the same `scribe login USER --auth AUTH` first: the client resolves the saved operation and does not create a second session. If the retry cannot succeed — the address was wrong, or you need another username — run `scribe login --discard`, then log in again. Discarding forgets only the local intent; a session the server may have issued for it stays unused until it expires. |
| `logout is pending; retry scribe logout` | A logout was sent but its answer was lost. Run `scribe logout` again; auth answers success for a credential it already revoked, so the retry completes. |
| `not logged in at SERVER` | There is no saved session for that server; nothing to log out. Check `--server` and `scribe whoami`. |
| Logout says the auth service is untrusted | Login with `--auth-ca FILE` saves a copy of that CA in the session directory (`auth-ca.pem`), so a plain `scribe logout` reuses it. If auth's CA has changed since, pass the current CA with `scribe logout --auth-ca FILE`. A trust failure does not leave a pending logout. A session that has already expired is removed without contacting auth. |
| `scribe-auth serve` refuses to start: `revision witness belongs to another auth incarnation` | The `--revision-file` given is not the one written for this data directory. Use the witness that `init`, `migrate`, or `restore` wrote for it. If the database really was replaced, run [`scribe-auth restore`](/docs/reference/cli/scribe-auth.md#restore). |
| `scribe-auth serve` refuses to start: `auth database is at revision R but its witness records W` | The auth database was restored from an older copy, which would revive revoked credentials. Do not replace or edit the witness; run [`scribe-auth restore`](/docs/reference/cli/scribe-auth.md#restore) from the backup, passing the current witness. See [auth backup and restore](/docs/migration-backup-restore.md#auth-backup-and-restore). |
| `the [auth] table is empty; set namespace, credential, tls_cert and tls_key, or remove the table` | The storage configuration has an `[auth]` table whose keys are all commented out. Set all four keys, or remove the table to serve plaintext. See [server configuration](/docs/server-usage.md#configuration-file). |
| `auth requires namespace, credential, TLS certificate and TLS key` | Only some `[auth]` settings resolved from the file and `--auth-*`/`--tls-*` flags. Supply all four. |
| `[auth].credential PATH is inside the data directory; keep secrets outside it` | Move the credential file (or `tls_key`) out of the data root: backups and host moves copy it. |
| `forbidden` | The message names the operation, the role it needs, and the repository as far as the client knows them (for example `submit needs the writer role in repository 'game'`). Ask a repository administrator for that grant; see [Access denied](/docs/troubleshooting.md#access-denied-or-repository-missing). |

> [!NOTE]
> Sessions and remembered trust are endpoint-scoped. Switching from direct storage to a gateway requires login/trust for the gateway address.

A desktop link that names a `tcp://` server or an `http://` web address on a
network host is refused (`a link cannot name a plaintext (tcp://) server on the
network; use tls://`); plaintext is accepted only for loopback addresses.

## Access denied or repository missing

Check grants using an **auth administrator credential**, not the user's session.
The [admin helper](/docs/tutorials/02-add-auth.md#4-connect-the-auth-administration-cli-and-add-users)
wraps `scribe-auth` with those operator credentials:

```sh
auth access --user USERNAME
auth grants list --repo studio/REPOSITORY_ID
```

`access` shows everything one account may do; replace `REPOSITORY_ID` with the
full ID to see every grant on one repository. Inaccessible repositories are
hidden from discovery. Reader allows sync; writer allows submit; maintainer
allows forced unlock; a namespace reader grant reads every repository in the
namespace and its inventory; namespace administration allows repository
creation. Auth administration alone grants no repository access. See [roles](/docs/auth.md#credentials-and-roles).

## Submit, sync, or locks block progress

| Symptom | Safe next step |
| --- | --- |
| Submit outcome unknown, exit 3 | Rerun `scribe submit` in the same workspace. Keep its pending state; the client resolves the original operation. |
| `base_mismatch` | For ordinary stale work, run `scribe resolve --update`, resolve conflicts, then submit. Finish or abort pending moves first. A grouped merge must be aborted and restarted against the new target; retain and reapply recovered edits. |
| `locked_by_other` / `lock_held` | Inspect `scribe status --remote` or `scribe lock list`; coordinate with the holder. Forced unlock is an explicit maintainer action. |
| `lock_required` | Inspect `scribe lock rules PATH`; `scribe reconcile` checks opened files and acquires required locks. Accepted `main` rules apply even on another branch. |
| Sync reports deferred opened paths | Submit the opened work, or revert only when you intend to discard it, then retry sync. Closing an editor does not clear Scribe's opened state. |
| File replacement fails with a sharing violation | Close applications holding the file and retry; keep the journal and recovery files. |
| Interrupted move or merge | Resume with `scribe move --continue` or `scribe merge --continue`. Use the matching abort command only when abandoning that work. |
| Cannot abort a pending move | Inspect move IDs and dependencies in status. Abort later dependent moves first; independent moves can be undone in any order. |
| Choosing one conflict also selects other paths | Rename/collision groups resolve atomically. Review all members; agreed names select contents at that destination, competing names select the complete layout. |
| `merge_incomplete` | The submission did not cover every path the source changed since the merge base. Abort the merge and plan it again (`scribe merge --abort`, then `scribe merge <branch>`); resolve each listed path, using `scribe resolve --ours PATH` when you mean to keep the target's version. |
| Ambiguous file origins or merge planning limit | Do not guess a rename. Abort and review an earlier explicit `--base` for ambiguous lineage; split an oversized merge by source revision. See the documented bounds. |
| `branch 'NAME' already exists (names are compared ignoring case and Unicode form)` | An active branch holds a colliding name: `Art` and `art` are the same branch name. Choose another name, or use the existing branch. |
| `scribe branch create` says the name is reserved by an unfinished branch creation | A client still holding the intent finishes it with `scribe branch --continue`. If that client is gone, an operator stops the server and releases the name with [`scribe-server staging-branches --abandon`](/docs/operations.md#unfinished-branch-creations). |
| `add`, `edit`, or `delete` says to finish conflict resolution | The merge in progress would submit that work in its integration commit. Finish it with `scribe merge --continue`, or discard it with `scribe merge --abort`, then open the path. |
| `.scribe/_state` has no `SCRIBE_FORMAT`: an interrupted workspace creation | Workspace creation stopped before its descriptor was published. Move that `.scribe/_state` directory aside (delete it only when it holds no local work), then run `scribe workspace create` again. |
| Old computer's workspace still appears on the server | Run `scribe workspace list --all-repos` from the new computer, then `scribe workspace remove WORKSPACE_ID --repo REPO`. This retires the registration and preserves the old computer's files. See [workspace administration](/docs/workspace-administration.md). |
| Workspace removal reports held locks | Inspect `scribe lock list --repo REPO`. A Maintainer can release the intended paths with `scribe lock release --repo REPO --force PATH`, then repeat removal. Without auth, any connected client can do this. |
| `too many new workspaces this hour; reuse or remove existing workspaces and retry later` | An authenticated server registers at most 64 new workspaces per user per hour (`limit_exceeded`). Reuse an existing working copy, or remove unused registrations with `scribe workspace remove`, and retry later. |
| `workspace_retired` | The workspace ID was permanently retired. Preserve its files and recovery copies; create a new working copy in a separate directory and copy over the needed local work. Re-registering the old ID cannot undo retirement. |
| `workspace list --all-repos --everyone` is forbidden | With auth, Maintainer permission is required in every selected repository. Use `--repo REPO` for a repository you administer, or omit `--everyone` to list your own registrations. Auth administration alone does not grant storage access. |
| Workspace removal lost its connection | Repeat the same command with the same per-user configuration directory. The client resolves the persisted operation outcome before sending a new request. |
| A selected repository mount is `unavailable` | The parent's own files still sync and its watermark advances. Grant access or exclude the mount, then rerun `scribe sync` to retry the child. |

Detailed procedures: [client workflows](/docs/client-usage.md), [conflict resolution](/docs/branches.md).

## Status is incomplete or work is still pending

Default `status` checks opened/workflow paths locally. Use `--scan` for unopened
changes and `--verify` for content hashing. Summary counts overlap because one
path can be modified, moved, and need reconciliation at once.

`--remote` adds current server observations. If a query fails, the CLI keeps the
local report, marks remote state `unavailable`, and exits 1; lock fields remain
unknown rather than showing a partially fetched lock list. Local inspection or
ignore-rule failures also exit 1 with available findings. Changes, conflicts,
and pending work alone do not make status fail.

A `resolved` merge still needs submission. A saved but unapplied choice needs
`merge --continue`. `needs_reconcile` records unchecked local intent; reconcile
checks it against the server without replacing local bytes. See
[status fields and workflow hints](/docs/client-usage.md#status-and-scanning).

## Find or download an older file

Ordinary per-file history follows recorded moves, forks, and merged ancestry.
Use `scribe log PATH --all-branches` to include related revisions on unmerged
branches. Keep the returned snapshot and pagination cursor together. Reusing
a deleted filename creates a separate file incarnation, not an automatic link
to the former file. See [history](/docs/client-usage.md#cross-branch-file-history).

For `scribe get PATH --revision N --to FILE`, use the path and branch that held
the file at that snapshot; a current name may not exist before its rename. The
destination must be new. The command requires the server and does not change
workspace state. If it reports a flush failure after publication, the verified
destination may already exist; inspect it before retrying. See
[historical downloads](/docs/client-usage.md#downloading-one-historical-file).

## Data root locked, full, or unhealthy

- **Locked:** stop its owner gracefully and wait. Do not delete `LOCK`; ownership
  is an OS lock, not the presence of the file.
- **Disk full:** stop the service, inspect `scribe-server usage --data-dir DIR`,
  and provide capacity. Do not delete stored objects or database files.
- **Integrity failure:** preserve the root and run offline `doctor`. Recover from
  a verified copy through the [maintenance procedure](/docs/migration-backup-restore.md).
- **Locator lost or pack entries not indexed:** `doctor` names the fix; run
  `scribe-server rebuild-locator --data-dir DIR` with the server stopped. It
  re-hashes every stored object and indexes only bytes that verify; skipped
  entries are listed and exit 1 means the repository still needs attention.
- **Database recovery required:** normal startup performs database recovery.
  Stop it before rerunning the read-only check. `doctor` never repairs data.

See [offline health checks](/docs/operations.md#routine-health-checks).

## Gateway cannot place or serve a repository

```sh
scribe-gateway --config ./scribe-gateway.toml status --json
```

Check backend health, observation freshness, free space, and the gateway
credential's expiry, namespace reader grant, and delegate grant
(`auth access --user scribe-gateway`). Restart the gateway after changing configured membership.
An instance that stays `unavailable` with an authorization error right after a
configuration change usually names the wrong `namespace`: the gateway's service
account must hold the namespace reader and delegate grants for the namespace
that instance was `auth-init`ed with,
and `scribe-ctl` refuses a gateway entry whose namespace differs from the
managed store it points at. A client error saying names are
`<namespace>/<name>` means the gateway serves several namespaces and has no
`default_namespace`; write the qualified name or configure a default.

> [!WARNING]
> Duplicate names, changed instance identities, and ownership conflicts require investigation. Do not erase the routing ledger or redirect a repository to a different root.

An unavailable backend leaves its repositories unavailable; the gateway is not
failover storage. See [placement and reconciliation](/docs/gateway-usage.md#placement-and-reconciliation).

## A remote cluster's scribe-ctl is missing or out of date

A remote command runs a scribe-ctl installed in the cluster's directory
there. When the remote has this machine's OS and architecture, any command
run with `--remote` installs this machine's own build first, so there is
nothing to fix. From a machine of another platform (a macOS laptop and a
Linux server, say) it cannot: the remote runs the scribe-ctl a `--source`
command or `bootstrap` last built there, and one built from an older
checkout does not know newer commands (`doctor`, for one: `remote.doctor`
fails). `doctor`'s `remote.ctl` names the fix. Refresh it without touching
the cluster, its services or a command running there:

```sh
scribe-ctl --config ./scribe-cluster.toml bootstrap --source <checkout> --plan
scribe-ctl --config ./scribe-cluster.toml bootstrap --source <checkout>
```

`--source` builds scribe-ctl there in Docker from a checkout of this
scribe-ctl's version; without docker on the remote, pass `--binaries DIR`
with this release built for the remote. See
[bootstrap](/docs/reference/cli/scribe-ctl.md#bootstrap).

## If you need help

Record the command, exit code, build version, endpoint, time, and relevant error
lines. Include whether the operation is pending and which roots are affected.
Share a minimal, reviewed excerpt; exclude credentials and private file contents.