# `scribe-ctl` CLI reference Scribe's operations tool. `scribe-ctl` sets up and operates named clusters from one definition file. It runs an auth service, storage services, and a gateway as supervised local processes (the native runtime), or as containers of one Docker Compose project it generates (the [docker runtime](/docs/reference/cli/scribe-ctl.md#docker-runtime), optionally with the web interface), and creates what they need: data roots, both server-certificate CAs, certificates, the auth pepper and revision witness, and every service credential. It starts services in dependency order and drains them in reverse order. Every mutating command discovers what exists, plans typed steps, and executes exactly that plan; `--plan` shows the plan and stops. `setup` provisions missing resources and is safe to rerun; `add auth` and `add gateway` grow a cluster by one service; `upgrade` moves it to a new release with backups, explicit migrations and rollback; `renew` replaces credentials and server certificates before they expire, proving each new one before the old goes, and rotates a CA through a signed handover; `up` requires a set-up cluster. With `--remote` (or a `[remote]` table) every command runs on the machine that holds the cluster, over ssh; `bootstrap` refreshes the scribe-ctl there without touching the cluster. Daemon mode detaches the supervisor but does not install a boot-time service or automatic restart policy. See the [cluster guide](/docs/cluster-usage.md) for the definition, variants, and lifecycle workflows. [scribe](/docs/reference/cli/scribe.md) | [scribe-server](/docs/reference/cli/scribe-server.md) | [scribe-auth](/docs/reference/cli/scribe-auth.md) | [scribe-gateway](/docs/reference/cli/scribe-gateway.md) | [scribe-runner](/docs/reference/cli/scribe-runner.md) ## Reading this reference Command synopses below come from this checkout’s built `--help` output. Angle brackets denote values, square brackets optional arguments, and `...` repeatable arguments. `[OPTIONS]` includes the global options below and any command-specific options. Global options work before or after a subcommand; parent-group options also apply to their child commands. Use `--` to end option parsing when passing a positional path that starts with a dash. Run `scribe-ctl --help` for the command list, `scribe-ctl COMMAND --help` for command help, and `scribe-ctl --version` for the build version. `scribe-ctl help COMMAND` also displays command help. The command sections below cover every public action. ## Help and version `-h` / `--help` works at every command level. `-V` / `--version` is a top-level option; place it directly after the executable name. ## Global options | Argument or option | Meaning | | --- | --- | | `--config ` | Cluster definition; defaults to scribe-cluster.toml beside this executable | | `--cluster ` | Named cluster; required when the file defines several (status and where show all) | | `--timings` | Report controller phase timings to stderr | | `--log-level ` | Diagnostic verbosity: off, error, warn, info, debug, or trace | | `--log-format ` | Diagnostic output format: text or json | | `--json` | Emit one JSON result on stdout; diagnostics remain on stderr | | `--remote ` | Run the command on the machine that holds the cluster, over the system ssh (keys, agent and ssh config only; never a password prompt): `[USER@]HOST:DIR`, `ssh://[USER@]HOST[:PORT]/DIR`, or `HOST` alone with the directory from the definition's `[remote]`. Overrides the cluster's `[remote]` table | ## Defaults and configuration Without `--config`, the controller reads `scribe-cluster.toml` beside its own executable. Relative paths in the definition resolve beside that file. With multiple clusters, select `--cluster NAME` for setup, lifecycle commands, and logs; `status` and `where` without a selector report all clusters. `setup` of a new definition names its cluster `local` unless `--cluster` says otherwise. `example` does not load configuration; it prints the [shipped template](https://github.com/Flying-Rat/scribe/blob/main/crates/scribe-ctl/example.toml). Services, their one-shot tools and the detached supervisor run from the cluster's active binary set, `/bin//`, which `/bin/current` selects (a file holding the build id, not a symlink). Until setup records the first set, they come from the definition's `[binaries]` paths, or from beside this `scribe-ctl`. Under docker the set is images instead, built from a checkout (see [Docker runtime](/docs/reference/cli/scribe-ctl.md#docker-runtime)). `scribe-ctl --build-info` answers with its own build and state format, like the service executables. Logging defaults to `warn`, on stderr. `--log-level` accepts `off`, `error`, `warn`, `info`, `debug`, or `trace`; `SCRIBE_LOG` also supplies a target filter. `--log-format` accepts `text` or `json`. There are no `--log-filter`, verbose, or quiet switches. `--timings` requires the default-enabled `timings` feature. `--json` selects command results independently of diagnostics. A cluster whose definition has a `[remote]` table (or any command given `--remote`) runs on that machine: see [Remote operation](/docs/reference/cli/scribe-ctl.md#remote-operation). `example` always runs here. ## Command index - [setup](/docs/reference/cli/scribe-ctl.md#setup) — Create or complete a cluster: definition, data roots, CAs, certificates, secrets and credentials - [add auth](/docs/reference/cli/scribe-ctl.md#add-auth) — Put auth in front of a storage-only cluster; repositories are kept - [add gateway](/docs/reference/cli/scribe-ctl.md#add-gateway) — Put a TLS gateway in front of an auth cluster's stores - [upgrade](/docs/reference/cli/scribe-ctl.md#upgrade) — Upgrade the cluster to a new release (executables, or images built from a checkout), with backups, explicit migrations, a health check and rollback - [prune](/docs/reference/cli/scribe-ctl.md#prune) — Remove what old releases left behind, and only what scribe-ctl made: older binary sets, their unused images, stray remote scribe-ctl builds, older upgrade backups - [renew](/docs/reference/cli/scribe-ctl.md#renew) — Replace every credential and server certificate that expires within 30 days, proving each before the old one goes; retire a CA rotation's old CA after its overlap; `--rotate-ca` starts a rotation; `--rekey` replaces servers' private keys - [adopt](/docs/reference/cli/scribe-ctl.md#adopt) — Take over a Docker deployment scribe-ctl did not create, its data and secrets left where they are, with a rollback to the old Compose projects - [access](/docs/reference/cli/scribe-ctl.md#access) — Show who can do what in auth: every account, its kind and state, and its privileges - [where](/docs/reference/cli/scribe-ctl.md#where) — List every file the cluster uses and what it is for, its endpoints, and the next commands - [example](/docs/reference/cli/scribe-ctl.md#example) — Print a complete example cluster definition to stdout - [up](/docs/reference/cli/scribe-ctl.md#up) — Start the cluster: auth, then storage, then the optional gateway (and under docker the web interface) - [down](/docs/reference/cli/scribe-ctl.md#down) — Cooperatively drain all services, including children of a lost supervisor - [status](/docs/reference/cli/scribe-ctl.md#status) — Show supervisor and service state and expiry warnings - [doctor](/docs/reference/cli/scribe-ctl.md#doctor) — Check the cluster where it runs, and this machine, without changing anything; exit 0, 1 (warnings) or 3 (failures) - [df](/docs/reference/cli/scribe-ctl.md#df) — Show live disk usage for every storage server - [repo list](/docs/reference/cli/scribe-ctl.md#repo-list) — List every repository with its storage server, head and size - [logs](/docs/reference/cli/scribe-ctl.md#logs) — Show the last lines of a service or supervisor log - [ssh](/docs/reference/cli/scribe-ctl.md#ssh) — Open a shell in the remote cluster's directory, or run a command there - [bootstrap](/docs/reference/cli/scribe-ctl.md#bootstrap) — Install or refresh the scribe-ctl that runs a remote cluster's commands, without touching the cluster ## setup Create or complete a cluster: definition, data roots, CAs, certificates, secrets and credentials. Resumable; safe to rerun ```text scribe-ctl setup [OPTIONS] ``` | Argument or option | Meaning | | --- | --- | | `--variant ` | What a new definition describes: `storage` (one plaintext server), `auth` (auth and one TLS server) or `gateway` (auth, one TLS server and a gateway). With an existing definition, it must match | | `--namespace ` | Auth namespace of the storage server in a new auth or gateway definition [default: studio] | | `--port-offset ` | Add this to every default port of a new definition (auth 7443, storage 7447, gateway 7557, web 8080 and 8082) [default: 0, or the next free multiple of 100 beside other clusters in the file] | | `--runtime ` | Where a new definition's services run: native (local processes) or docker (a Compose project scribe-ctl generates) [default: native] | | `--web` | With a new docker definition: also run the web interface (apps/web) in the cluster's Compose project | | `--source ` | Under docker: the Scribe checkout (of this scribe-ctl's version) to build the images from, tagged with this scribe-ctl's build. Also builds with --plan; nothing in the cluster changes | | `--binaries ` | With --remote: this release's executables built for the remote machine (scribe-ctl, scribe-auth, scribe-server, scribe-gateway), pushed and installed there beside its scribe-ctl, which setup records. Not needed when this machine's OS and architecture match the remote's (this scribe-ctl's own directory is pushed) or under docker with --source (scribe-ctl is built there) | | `--plan` | Show the plan and run every read-only check, without writing anything or taking a lock | When the file does not define the cluster, `--variant` is required and setup writes the definition first (with an explanatory comment on every setting), plus `scribe-gateway.toml` for the gateway variant. Paths are relative to the definition; services listen on loopback. A file that already defines other clusters is rewritten canonically, the previous one kept as `.bak`; the new cluster's paths start with its name and its ports move clear of the others. When the file defines the cluster, setup uses it as written and refuses a `--variant` it does not match; `--namespace` and `--port-offset` are refused, and so are a `--runtime` other than the definition's (setup never moves a cluster between runtimes) and `--web` for a definition without `[web]`. Setup then creates whatever is missing, in dependency order: both CAs (`certificates.ca_dir` for storage and gateway, `certificates.auth_ca_dir` for auth) and each server certificate when `certificates.generate_missing` is on; the gateway ledger; auth with its pepper, revision witness and bootstrap administrator credential (`scribe-auth init`); each store, bound to its auth namespace. With auth running (started temporarily when the cluster is stopped) it then ensures: - each authenticated store is registered and holds a live server credential; - for a TLS gateway, the `scribe-gateway` service account with namespace reader and delegate grants in every namespace it fronts, and its credential at the gateway's `[tls] credential` path; - for every `monitor_credential`, the `scribe-monitor` service account with a namespace reader grant and that credential; - every user in `accounts_file`. A credential is issued when its file is missing and kept otherwise, also near or past its expiry: setup never replaces one. Its plan names each service credential within 30 days of expiry and points to [renew](/docs/reference/cli/scribe-ctl.md#renew), which replaces it. A file of another holder, or a revoked one, is refused. Last, setup writes `CLUSTER-INFO.txt` beside the definition (see [where](/docs/reference/cli/scribe-ctl.md#where)). Setup also records the executables the cluster runs, so that an [upgrade](/docs/reference/cli/scribe-ctl.md#upgrade) has a set to return to. When the state directory has no `bin/current` yet, it takes them from the definition's `[binaries]` paths, or from beside this `scribe-ctl` (`scribe-ctl` itself included), checks that they are one release for this machine (`--build-info`), copies them side by side into `/bin//` and writes `bin/current`. From then on every service, tool and the detached supervisor starts from that set; `[binaries]` is not read again. A cluster set up before sets were recorded gets one when setup runs again, even while it runs. Before changing anything setup runs its checks: the definition, binaries (the recorded set must have every executable the cluster runs), private permissions on existing secrets and an existing auth data root (auth refuses to start from a root other accounts can enter; the check names the `chmod 700` that fixes it), the accounts file, whether the cluster runs, free listen ports, certificate and data-root consistency. A failed check refuses the whole run. Existing state setup did not make, such as a partial PEM pair or a data root without its format marker, is refused rather than replaced. While the cluster runs, setup performs only the identity steps through the running auth (for example new accounts). Setup records each step in a journal in the state directory before and after running it. An interrupted setup resumes when run again: a step it had started is redone, and its partial output is first moved aside beside the original as `.interrupted-`, never deleted. If an interrupted run left auth running, `status` shows it as unmanaged; run `down` before setup again. `--plan` prints the same plan (checks, steps with the reason for each, notes) and exits 1 when a check fails. It writes nothing, takes no lock and starts nothing. For example, for a new gateway cluster: ```text $ scribe-ctl --config ./scribe-cluster.toml setup --variant gateway --plan Scribe · local · setup plan · gateway variant · native runtime Definition /srv/scribe/scribe-cluster.toml Paths below are relative to its directory. ENDPOINTS auth tls://127.0.0.1:7443 storage-a tls://127.0.0.1:7447 gateway tls://127.0.0.1:7557 CHECKS ok definition setup writes scribe-cluster.toml (gateway variant) ok binaries scribe-ctl, scribe-auth, scribe-server, scribe-gateway 0.1.0; setup records them as the first binary set in cluster-state/local/bin/0.1.0 ok permissions 0 existing secret files are private ok cluster stopped ok ports 3 listen addresses are free ok certificates consistent ok data roots consistent STEPS 1. Write the cluster definition scribe-cluster.toml and scribe-gateway.toml no definition exists yet 2. Install scribe-ctl, scribe-auth, scribe-server, scribe-gateway 0.1.0 side by side in cluster-state/local/bin/0.1.0 (from releases/0.1.0) the cluster's executables, kept side by side so an upgrade can return to them 3. Record 0.1.0 as the active set (bin/current) the runtime starts services from the set bin/current names 4. Create the storage and gateway CA in pki/ca a server certificate needs this CA 5. Create the auth CA in pki/auth-ca a server certificate needs this CA 6. Issue auth's server certificate pki/auth/server.pem no certificate yet 7. Issue storage-a's server certificate pki/storage-a/server.pem no certificate yet 8. Issue gateway's server certificate pki/gateway/server.pem no certificate yet 9. Initialize the gateway ledger data/gateway no gateway ledger yet 10. Initialize auth data/auth with its pepper, revision witness and administrator credential no auth database yet 11. Initialize storage storage-a in data/storage-a no repository store yet 12. Bind storage storage-a to auth namespace studio the store is not bound to its auth namespace yet 13. Register storage storage-a with auth auth must know the server before it issues its credential 14. Ensure storage-a's server credential secrets/storage-a.credential issued when missing; an existing one is kept (`renew` replaces it before it expires) 15. Ensure service account scribe-gateway reads and acts for users in studio the gateway reads inventory and acts for users in the namespaces it fronts 16. Ensure scribe-gateway's credential secrets/gateway.credential issued when missing; an existing one is kept (`renew` replaces it before it expires) 17. Ensure service account scribe-monitor reads in studio scribe-ctl reads storage inventory (df, repo list) as this account 18. Ensure scribe-monitor's credential secrets/monitor.credential issued when missing; an existing one is kept (`renew` replaces it before it expires) 19. Write the cluster summary CLUSTER-INFO.txt every file the cluster uses, its endpoints and the next commands NOTES - Auth runs temporarily for steps 13-18 and is stopped afterwards. - The CA private keys (ca-key.pem in pki/ca and pki/auth-ca) are the keys to the cluster: back them up offline and never share them. They are needed only to issue and renew server certificates and to rotate the CA (`renew --ca-dir DIR` signs with a key kept elsewhere). Plan only: nothing was written and no lock was taken. Run setup without --plan to apply it. ``` ## Docker runtime With `runtime = "docker"` (`setup --runtime docker`) every service is a container of one Compose project that `scribe-ctl` generates from the definition and drives with the `docker` CLI (Docker Engine with Compose v2 and `--wait`; no Docker API client). The commands and plans are the same as natively; only where services run changes. **Images.** `setup --source DIR` and `upgrade --source DIR` build the images from a Scribe checkout with the Dockerfiles in `docker/` (and, with the web interface, every image `apps/web/compose.yaml` builds), before anything is planned, `--plan` included: every check runs against them, and building changes nothing in the cluster. They are tagged with this `scribe-ctl`'s build (`scribe/auth:BUILD`, `scribe/storage:BUILD`, `scribe/gateway:BUILD`, `scribe/web-SERVICE:BUILD`; the build is passed as `SCRIBE_BUILD_ID`), and the checkout must be this `scribe-ctl`'s version: a release's own controller runs it, so build `scribe-ctl` from the same checkout (set `SCRIBE_BUILD_ID` for a build of the same version) and run that; `scripts/install.sh --setup` and `--upgrade` do both, naming the build by the commit. A build id that cannot tag an image (`+`) is refused. Each image answers `--build-info` from a throwaway container (`docker run --rm --network none`), checked like executables: one release, built for the Docker engine's OS and architecture (`docker version`). The set is recorded as natively, `/bin//SET.json` (with the web interface's Compose file, seccomp profile and service list under `web/`), and `bin/current` selects the images the project runs. **The project.** Before every start `scribe-ctl` writes `/docker/`: `compose.yaml` (JSON, which Compose reads as YAML), the gateway's configuration as its container reads it (`gateway.toml`), and with the web interface `web.yaml` (the active set's copy of `apps/web/compose.yaml`), its `.env` and the public CAs in `trust/`. Change the definition, not these files. The project is named `scribe-CLUSTER-HASH`, so two definitions with a cluster of the same name never share containers; `where` prints the `docker compose` command for it. Containers see fixed paths, never the host's: a data root is `/data`, a service certificate `/pki/server.pem` and `/pki/server-key.pem`, a credential under `/secrets/`, auth's witness directory `/witness`. Host paths appear only as bind-mount sources (Windows' `\\?\` prefixes removed), which must exist: Compose never creates one. Every container runs as the owner and group of the definition's directory (not on Windows, where the images' own user applies), with a read-only root, no capabilities and `no-new-privileges`, listens on its service's port on every interface of its own network namespace and publishes it on the definition's `listen` address only. Its health is its readiness file. A plaintext store therefore passes `--allow-non-loopback` inside its container: other containers of the project (the web interface) reach it on the project network, while the host sees only the published loopback address. Containers reach each other by service name on the project network: `tls://auth:7443`, `tls://storage-a:7447`, `tls://gateway:7557`. Certificates name the service too, and the credentials of containers (each store's, the gateway's, the web reader's) carry auth's service address, while the administrator's and the inventory credential carry its published address, since `scribe-ctl` and people use them from the host. The gateway's container configuration names each store the cluster runs by its service name; every other setting is copied from the definition's gateway file. Docker needs of the definition: a fixed port for every service; no `[binaries]`; and the revision witness in a directory of its own (the pepper may share it), because auth's container mounts that directory writable to replace the witness. Setup's docker layout uses `secrets/auth/`. **Tools and lifecycle.** One-shot tools (`init`, `auth-init`, `migrate`) run as `docker run --rm --network none` of the set's image with exactly the mounts they need. Identities are provisioned from the host through auth's published port, as natively; setup starts auth alone (`docker compose up -d --wait auth`) while it needs it. `up` is `docker compose up -d --wait --no-build --remove-orphans` (the gateway, when `require_healthy`, is then waited for until it reports fresh inventory); `down` is `docker compose stop` (containers and their logs are kept, data is never touched); `status` reads `docker compose ps`; `logs SERVICE` is `docker compose logs --tail N SERVICE` for any service of the project. `up --foreground` starts detached as well and stops the project on Ctrl-C. ### Web interface `setup --runtime docker --web` adds a `[web]` table and runs the web interface (apps/web) in the cluster's project, with its own services and networks as `apps/web/compose.yaml` defines them. `scribe-ctl` adds only the wiring: its images; the cluster network for the services that reach Scribe (those on apps/web's `upstream` network); the public CAs (`trust/ca.pem` for storage and the gateway, `trust/auth-ca.pem` for auth); the reader credential's directory; the database's files in `[web] data_dir` (run as the directory's owner, so upgrades can back them up); and `jwt_key` and `surreal_password`, which setup writes (32 random bytes, base64, each atomically; a database whose password file is gone is refused, never given a new one). The `.env` points the web services at the gateway (or the first store) and auth by service name, and the Clone menu at the published addresses. With auth, setup creates the `scribe-web` service account (a namespace reader in every namespace of the cluster) and its credential. A new web database is prepared by the worker image's `migrate.js`, which the web services require before they start. The web interface's own TLS certificate is yours: put `fullchain.pem` and `key.pem` in `[web] tls_dir` and set `origin` to the `https://` address; the next start serves HTTPS on `listen` and redirects HTTP on `redirect_port`. Without them it serves plain HTTP; keep it on loopback. `where` says where the files go. See the [web interface guide](/docs/web-interface.md#run-it-with-scribe-ctl). For a new gateway cluster with the web interface: ```text $ scribe-ctl --config ./scribe-cluster.toml setup --variant gateway --runtime docker --web --source /srv/scribe-src --plan Building scribe/auth:0.1.0 from /srv/scribe-src (the first build takes a while)… Building scribe/storage:0.1.0 from /srv/scribe-src (the first build takes a while)… Building scribe/gateway:0.1.0 from /srv/scribe-src (the first build takes a while)… Building the web interface's images (api, …) from /srv/scribe-src/apps/web… Scribe · local · setup plan · gateway variant · docker runtime Definition /srv/scribe/scribe-cluster.toml Paths below are relative to its directory. ENDPOINTS auth tls://127.0.0.1:7443 storage-a tls://127.0.0.1:7447 gateway tls://127.0.0.1:7557 web http://localhost:8080 CHECKS ok definition setup writes scribe-cluster.toml (gateway variant) ok binaries scribe-auth, scribe-server, scribe-gateway 0.1.0; setup records them as the first image set in cluster-state/local/bin/0.1.0 ok permissions 0 existing secret files are private FAIL cluster some of the cluster's containers run (storage-a, nginx, api, worker, database) but not auth, gateway; run `scribe-ctl --config /srv/scribe/scribe-cluster.toml down` first ok certificates consistent ok data roots consistent STEPS 1. Write the cluster definition scribe-cluster.toml and scribe-gateway.toml no definition exists yet 2. Record the images scribe/auth:0.1.0, scribe/storage:0.1.0, scribe/gateway:0.1.0 and the web interface's of 0.1.0 as a set in cluster-state/local/bin/0.1.0 the cluster's executables, kept side by side so an upgrade can return to them 3. Record 0.1.0 as the active set (bin/current) the runtime starts services from the set bin/current names 4. Create the storage and gateway CA in pki/ca a server certificate needs this CA 5. Create the auth CA in pki/auth-ca a server certificate needs this CA 6. Issue auth's server certificate pki/auth/server.pem no certificate yet 7. Issue storage-a's server certificate pki/storage-a/server.pem no certificate yet 8. Issue gateway's server certificate pki/gateway/server.pem no certificate yet 9. Initialize the gateway ledger data/gateway no gateway ledger yet 10. Initialize auth data/auth with its pepper, revision witness and administrator credential no auth database yet 11. Initialize storage storage-a in data/storage-a no repository store yet 12. Bind storage storage-a to auth namespace studio the store is not bound to its auth namespace yet 13. Create the web interface's missing secrets secrets/web/jwt.key and secrets/web/surreal.password the web services sign sessions and open their database with them 14. Register storage storage-a with auth auth must know the server before it issues its credential 15. Ensure storage-a's server credential secrets/storage-a.credential issued when missing; an existing one is kept (`renew` replaces it before it expires) 16. Ensure service account scribe-gateway reads and acts for users in studio the gateway reads inventory and acts for users in the namespaces it fronts 17. Ensure scribe-gateway's credential secrets/gateway.credential issued when missing; an existing one is kept (`renew` replaces it before it expires) 18. Ensure service account scribe-monitor reads in studio scribe-ctl reads storage inventory (df, repo list) as this account 19. Ensure scribe-monitor's credential secrets/monitor.credential issued when missing; an existing one is kept (`renew` replaces it before it expires) 20. Ensure service account scribe-web reads in studio the web services read repositories as this account 21. Ensure scribe-web's credential secrets/web-reader/scribe-web.credential issued when missing; an existing one is kept (`renew` replaces it before it expires) 22. Prepare the web database data/web (migrate.js on the empty store) no web database yet 23. Write the cluster summary CLUSTER-INFO.txt every file the cluster uses, its endpoints and the next commands NOTES - Auth runs temporarily for steps 14-21 and is stopped afterwards. - The CA private keys (ca-key.pem in pki/ca and pki/auth-ca) are the keys to the cluster: back them up offline and never share them. They are needed only to issue and renew server certificates and to rotate the CA (`renew --ca-dir DIR` signs with a key kept elsewhere). setup cannot run until every check passes. Plan only: nothing was written and no lock was taken. Run setup without --plan to apply it. ``` ## Remote operation A cluster can live on another machine (a VM, a server) while `scribe-ctl` runs here. Give its definition a `[remote]` table, or pass `--remote TARGET` to any command; the definition file here stays the source of truth, and secrets stay there. ```toml [clusters.prod.remote] host = "scribe-host" # a host name, an address, or an alias from ~/.ssh/config user = "deploy" # optional: else the ssh config's, or yours port = 22 # optional directory = "/srv/scribe" # holds the cluster there; absolute, or relative to the login's home ``` `--remote` takes `[USER@]HOST:DIR`, `[USER@][IPV6]:DIR`, `ssh://[USER@]HOST[:PORT]/DIR` (`/~/DIR` for one in the home directory), or a bare `HOST` whose directory comes from the definition's `[remote]`. It replaces the table for that command (user and port included). Host, user and directory are validated before anything runs: nothing that ssh could read as an option reaches it, and every word passed to the remote shell is quoted. For each command, `scribe-ctl` here: 1. connects once with the system `ssh` in batch mode (`-o BatchMode=yes`, a 30 s connect timeout, keepalives every 15 s so a silently dropped connection ends within about a minute) and keeps that connection open as a master (`ControlMaster`, its socket in a private directory under `/tmp`) that every later ssh, tar and rsync of the command goes through: a command makes one connection however many steps it has. The master ends with the command, on success, error or Ctrl-C. Options on the command line win over the ssh config, so a `ControlMaster` set there does not interfere. On Windows, whose OpenSSH has no connection sharing, each step connects on its own. A first connection the remote times out, refuses or drops (a busy machine, a connection limit, a firewall throttling bursts) is tried four more times, after 10, 30, 60 and 90 s: ```text ssh connect to scribe-host failed (ssh: connect to host scribe-host port 22: Operation timed out); retrying in 10 s (2/5)… ``` Authentication, host-key and name failures are not retried. Then it probes the remote: its OS and architecture (`uname`), rsync and docker, and the scribe-ctl builds installed in the cluster's directory; 2. pushes what the command needs: from the checkout of `--source DIR`, only what the image builds read, to `upload/source` (the Rust workspace, the Dockerfiles and `apps/web`, each without the paths its `.dockerignore` leaves out, and never `.git`, `target*`, `node_modules`, `.env` files, `*.credential`, `*.pepper`, `secrets/`, `pki/` or `cluster-state/`; see [what a push carries](/docs/cluster-usage.md#remote-operation)), and the executables of `--binaries DIR` to `upload/binaries`, which must run on the remote (its scribe-ctl answers `--build-info` there for the remote's OS and architecture). rsync 3 is used when both ends have it (only changes travel; everything else there is deleted); otherwise a tar stream through ssh, unpacked beside the previous upload and swapped in; 3. chooses the scribe-ctl that runs the command there, `bin//scribe-ctl` in the cluster's directory, installing it when it is missing: - with `--source`, the one built from exactly the pushed checkout, in Docker there (`docker/Dockerfile.ctl`, exported as a file). Its build is `-src.`: the first 12 hex digits of a BLAKE3 digest of the pushed files' paths and contents. Any change to the checkout builds a new scribe-ctl there, and nothing else does (modification times, the order files are listed in); the images it builds carry the same build. This needs docker on the remote; - otherwise one rule, which `doctor` follows too: 1. this scribe-ctl's exact build (same version, build and target) when it is there; on another OS or architecture only when its build id names a build (`SCRIBE_BUILD_ID`), since a build that is only the version may come from any source; 2. else this build, installed: from the pushed `--binaries` when that is this release, or from this scribe-ctl's own directory (every executable of this release beside it) when both machines have the same OS and architecture; 3. else the one `bin/controller` names when it is this version for the remote's machine: the newest installed there, by [bootstrap](/docs/reference/cli/scribe-ctl.md#bootstrap) or by the last command that installed one (`Running the remote's scribe-ctl 0.1.0 (build 0.1.0-src.3fa9c2e1b7d0) (bin/controller): this 0.1.0 is built for macos/aarch64`); 4. else the one `bin/current` names, on the same terms; 5. else this build after all, when it is there. Otherwise it stops and names [bootstrap](/docs/reference/cli/scribe-ctl.md#bootstrap). It installs into a new directory, moves an existing one aside, checks the installed scribe-ctl answers as that build, and only then names it in `bin/controller` and `bin/current`. Never system-wide. A remote whose `bin/current` names a newer version is refused: use that release here too. Builds of one version are one release, as everywhere: upgrading from one to another is allowed; 4. pushes the cluster's definition: the remote copy (`DIR/`) gets this file's version of the cluster without its `[remote]` table, other clusters there are kept, and a differing copy is kept as `.bak`. A generated gateway configuration beside the definition goes along. Nothing is pushed for a cluster this file does not define yet (setup creates it there); 5. runs the same command there, `cd DIR && exec bin//scribe-ctl --config [--cluster NAME] [--json] … COMMAND …` with the build chosen above, the pushed directories named as they are there. Discovery, planning, the command lock, the journal and execution all happen on the remote. Its stdout and stderr stream through unchanged (`--json` too), with a terminal when this one has one. An image build prints nothing until it ends, so while one runs the remote scribe-ctl says `still building IMAGE (N min)` once a minute. The remote command outlives its connection: when it drops (or this terminal closes) the scribe-ctl there carries on to the end, or through its rollback, with its output lost; `status` shows how it went, and running the command again resumes whatever it left pending. Ctrl-C asks it to stop at its next safe point, as locally: through the remote terminal, or without one (`--json`, redirected output) as `kill -INT` of the process id it announces, sent through the connection; a second Ctrl-C leaves it running and returns; 6. after a successful `setup` or `add`, brings the cluster's definition back into this file with its `[remote]` table (the previous file kept as `.bak`), and after `setup`, `add` or `upgrade` brings the summary back as `CLUSTER-INFO-.txt` beside it (its paths are the remote's). Nothing else comes back: CA keys, the pepper, credentials and data stay there. The exit code is the remote scribe-ctl's; 255 means ssh itself failed. ssh never prompts: a host it cannot reach without a password or a host-key confirmation fails with what to do: ```text $ scribe-ctl --config ./scribe-cluster.toml --remote deploy@unreachable-vm:/srv/scribe status Checking deploy@unreachable-vm:/srv/scribe over ssh… Scribe · status failed cannot reach deploy@unreachable-vm:/srv/scribe with ssh (ssh: Could not resolve hostname unreachable-vm: nodename nor servname provided, or not known). scribe-ctl runs the system ssh in batch mode: it uses your keys, agent and ~/.ssh/config and never asks for a password or to accept a host key. Make `ssh deploy@unreachable-vm true` work without a prompt (install a key with ssh-copy-id, connect once to accept the host key, or add a Host alias to ~/.ssh/config) ``` Setting up a cluster on a machine of the same OS and architecture as this one (here with a relative directory, in the login's home): ```text $ scribe-ctl --config ./scribe-cluster.toml --remote deploy@scribe-host:scribe setup --variant auth Checking deploy@scribe-host:scribe over ssh… Installing scribe-ctl 0.1.0 on the remote with this machine's scribe-ctl, scribe-auth, scribe-server, scribe-gateway… Running `setup` on deploy@scribe-host:scribe… local · [1/15] Write the cluster definition scribe-cluster.toml… … Scribe · local · Setup complete wrote scribe-cluster.toml installed 0.1.0 in cluster-state/local/bin/0.1.0 … wrote CLUSTER-INFO.txt Every file, endpoint and next command: /home/deploy/scribe/CLUSTER-INFO.txt (or `where`). Next: scribe-ctl --config /home/deploy/scribe/scribe-cluster.toml up Wrote cluster local into /home/me/ops/scribe-cluster.toml with its [remote] table… Brought the cluster summary back as /home/me/ops/CLUSTER-INFO-local.txt (paths in it are on the remote)… ``` The remote copy of the definition has this file's name. From then on the table is in this file and every command goes there without `--remote`: ```text $ scribe-ctl --config ./scribe-cluster.toml up Checking deploy@scribe-host:scribe over ssh… Running `up` on deploy@scribe-host:scribe… local · Starting cluster… Scribe · local · running SERVICE STATE PID ENDPOINT auth running 75409 tls://127.0.0.1:7443 storage-a running 75421 tls://127.0.0.1:7447 Logs /home/deploy/scribe/cluster-state/local/logs Binaries 0.1.0 · /home/deploy/scribe/cluster-state/local/bin/0.1.0 $ scribe-ctl --config ./scribe-cluster.toml logs nonexistent; echo "exit $?" Checking deploy@scribe-host:scribe over ssh… Running `logs` on deploy@scribe-host:scribe… Scribe · logs failed unknown service exit 1 ``` Output of the remote command names the remote's paths (its `Next:` line is the remote's command line too). Services listen where the definition says, on the remote: setup's layouts listen on loopback, which is the remote's own. For clients elsewhere, write the definition here first (`scribe-ctl example`, or a setup `--plan` to start from), with the listen addresses and certificate names (`[tls] dns`/`ip`) clients use and the `[remote]` table, and run setup: it pushes that definition and sets it up as written. `--plan` runs remotely like any command; it pushes the definition and may install scribe-ctl in `bin/` first, but changes nothing of the cluster. With several clusters in this file and no `--cluster`, `status` and `where` cover the local ones and name each remote one with the command that asks it. Every command works remotely except `example` (local) and the hidden supervisor; for a shell there, see [ssh](/docs/reference/cli/scribe-ctl.md#ssh). On the remote, the cluster's directory holds the definition's copy, `bin//` (a scribe-ctl per release that ran there, with the service executables it was installed with: native setup records those, so a native cluster's first binary set comes from there), `bin/controller` (the newest scribe-ctl installed there), `bin/current` (the one the last command that installed one ran with; for a `--source` setup or upgrade, the build its images carry), `upload/source` and `upload/binaries`, `upload/controller/` (what [bootstrap](/docs/reference/cli/scribe-ctl.md#bootstrap) pushed), and whatever the definition names (data, secrets, pki, state). These `bin/` files are the remote directory's, not the cluster's active set in `/bin/current`. Remove old `bin/` directories by hand. ## add auth Put auth in front of a storage-only cluster: CAs, certificates, the auth database, and TLS with sign-in on every store. Repositories are kept. The cluster must be stopped ```text scribe-ctl add auth [OPTIONS] ``` | Argument or option | Meaning | | --- | --- | | `--namespace ` | Auth namespace every store joins [default: studio] | | `--listen ` | Where auth listens [default: 127.0.0.1 on 7443, moved by as much as the first store's port is from 7447] | | `--plan` | Show the plan with the definition diff and run every read-only check, without writing anything or taking a lock | `add auth` converts a storage-only cluster (the `storage` variant, or any definition of plaintext stores with no auth and no gateway) into the `auth` variant. It rewrites the definition canonically, keeping the previous file as `.bak`: an `[auth]` table (data, pepper, revision witness, administrator credential and certificate under `data/`, `secrets/` and `pki/`), a `[certificates]` table when there is none (an existing one is kept as written), and on every store `auth = true`, the namespace, its credential, the `scribe-monitor` inventory credential and its certificate. A store keeps its address; only the scheme changes from `tcp://` to `tls://`. In a file that defines several clusters the new paths start with the cluster's name. Then it provisions exactly what [setup](/docs/reference/cli/scribe-ctl.md#setup) would for that definition: both CAs if missing, the auth and store certificates, the auth database, each store bound to its namespace (`scribe-server auth-init`) in place, its registration and server credential, and the inventory account. Data roots are never initialized again or deleted: repositories and their history stay, and the storage data needs no copy. Once bound, a store refuses to start without auth, so the cluster must be stopped for the whole change; while it runs, the plan fails its `cluster` check and nothing is changed. Run `down`, then `add auth`, then `up`. What changes for people is listed under AFTERWARDS: clients switch to `tls://` with `--tls-ca pki/ca/ca.pem` and log in through auth with `--auth-ca pki/auth-ca/ca.pem`; nobody has access until an administrator adds people and grants (the Next commands in `CLUSTER-INFO.txt`); and a workspace created before auth has no signed-in owner, so it still syncs but cannot submit: create a new workspace after logging in. `add auth` is resumable like setup, with the same journal (operation `add auth`), set-aside rule and refusal of state it did not make. A rerun after it completed, or on a cluster that already has auth, changes nothing and says `nothing to add`; one whose auth is incomplete without an interrupted add is refused with a pointer to `setup`. `--namespace` and `--listen` are refused once auth exists, and so is a custom topology (for example plaintext and TLS stores side by side). A listen port another service of the file uses is refused. For a storage cluster set up with default ports: ```text $ scribe-ctl --config ./scribe-cluster.toml add auth --plan Scribe · local · add auth plan · auth variant · native runtime Definition /srv/scribe/scribe-cluster.toml Paths below are relative to its directory. ENDPOINTS auth tls://127.0.0.1:7443 storage-a tls://127.0.0.1:7447 CHECKS ok definition add auth rewrites scribe-cluster.toml with auth ok binaries scribe-ctl, scribe-auth, scribe-server 0.1.0, the active set in cluster-state/local/bin/0.1.0 ok permissions 0 existing secret files are private ok cluster stopped ok ports 2 listen addresses are free ok certificates consistent ok data roots consistent DEFINITION CHANGES @ scribe-cluster.toml [clusters.local] # What setup builds: storage, auth or gateway. Setup refuses services that do not match it. - variant = "storage" + variant = "auth" # Supervisor state, service logs and setup's journal. Never share it between clusters. state_dir = "cluster-state/local" … startup_timeout_secs = 30 + # The two server-certificate CAs. + [clusters.local.certificates] + # Setup may create missing CAs and server certificates. Once they exist, move the + # CA private keys (ca-key.pem) offline and set this to false. + generate_missing = true + # Signs storage and gateway certificates. + ca_dir = "pki/ca" + # Signs only auth's certificate, so no storage or gateway key can pass as auth. + auth_ca_dir = "pki/auth-ca" + + # Accounts, credentials and access decisions. TLS only. + [clusters.local.auth] + # A fixed port: credentials name this address. + listen = "127.0.0.1:7443" + data_dir = "data/auth" + # Written by setup and needed by every auth start. Back it up apart from the data. + pepper_file = "secrets/auth.pepper" + # Revision witness: keep it outside the data directory and its backups. + revision_file = "secrets/auth.revision" + # The bootstrap administrator's credential, written by setup. Keep it private. + admin_credential = "secrets/admin.credential" + + # Server certificate and key; setup issues them when generate_missing is on. + [clusters.local.auth.tls] + cert = "pki/auth/server.pem" + # Private key: never share it. + key = "pki/auth/server-key.pem" + # A storage server. [[clusters.local.storage]] … data_dir = "data/storage-a" # Clients sign in through auth; requires TLS. - auth = false + auth = true + # The auth namespace this server's repositories belong to. + namespace = "studio" + # This server's own credential, issued by setup. Keep it private. + credential = "secrets/storage-a.credential" + # A namespace reader's credential; scribe-ctl uses it for df, repo list and status --storage. + monitor_credential = "secrets/monitor.credential" + + # Server certificate and key; setup issues them when generate_missing is on. + [clusters.local.storage.tls] + cert = "pki/storage-a/server.pem" + # Private key: never share it. + key = "pki/storage-a/server-key.pem" STEPS 1. Rewrite the definition scribe-cluster.toml with auth, keeping the previous one as scribe-cluster.toml.bak the definition has no auth yet; it is rewritten canonically 2. Create the storage and gateway CA in pki/ca a server certificate needs this CA 3. Create the auth CA in pki/auth-ca a server certificate needs this CA 4. Issue auth's server certificate pki/auth/server.pem no certificate yet 5. Issue storage-a's server certificate pki/storage-a/server.pem no certificate yet 6. Initialize auth data/auth with its pepper, revision witness and administrator credential no auth database yet 7. Bind storage storage-a to auth namespace studio the store is not bound to its auth namespace yet 8. Register storage storage-a with auth auth must know the server before it issues its credential 9. Ensure storage-a's server credential secrets/storage-a.credential issued when missing; an existing one is kept (`renew` replaces it before it expires) 10. Ensure service account scribe-monitor reads in studio scribe-ctl reads storage inventory (df, repo list) as this account 11. Ensure scribe-monitor's credential secrets/monitor.credential issued when missing; an existing one is kept (`renew` replaces it before it expires) 12. Write the cluster summary CLUSTER-INFO.txt every file the cluster uses, its endpoints and the next commands NOTES - The definition file is rewritten in canonical form; the previous one is kept as scribe-cluster.toml.bak. - Auth runs temporarily for steps 8-11 and is stopped afterwards. - The CA private keys (ca-key.pem in pki/ca and pki/auth-ca) are the keys to the cluster: back them up offline and never share them. They are needed only to issue and renew server certificates and to rotate the CA (`renew --ca-dir DIR` signs with a key kept elsewhere). - Storage data and repositories are kept: every store is bound to namespace studio in place, never re-initialized. Once bound, a store no longer starts without auth. AFTERWARDS - Start it with `scribe-ctl --config /srv/scribe/scribe-cluster.toml up`: auth first, then storage. - Clients switch from tcp:// to tls://127.0.0.1:7447 (trusting pki/ca/ca.pem with --tls-ca) and log in through auth at tls://127.0.0.1:7443 (trusting pki/auth-ca/ca.pem with --auth-ca); plaintext connections are refused from now on. - Nobody has access yet: add people and grants with scribe-auth and the administrator credential secrets/admin.credential (the Next commands in CLUSTER-INFO.txt). - Workspaces created before auth have no signed-in owner: they still sync, but submitting needs a new workspace (scribe workspace create) made after logging in; bring unsubmitted work over by hand. - Give clients pki/ca/ca.pem and pki/auth-ca/ca.pem; keep the CA private keys (ca-key.pem beside them) offline. Plan only: nothing was written and no lock was taken. Run add auth without --plan to apply it. ``` ## add gateway Put a TLS gateway in front of an auth cluster's stores, with the scribe-gateway service account and its credential. The cluster must be stopped ```text scribe-ctl add gateway [OPTIONS] ``` | Argument or option | Meaning | | --- | --- | | `--listen ` | Where the gateway listens [default: 127.0.0.1 on 7557, moved by as much as auth's port is from 7443] | | `--plan` | Show the plan with the definition diff and run every read-only check, without writing anything or taking a lock | `add gateway` turns the `auth` variant (auth and TLS stores, no gateway) into the `gateway` variant. It writes `scribe-gateway.toml` beside the definition (one `[[instances]]` entry per store, in the store's namespace, trusting the cluster CA) and rewrites the definition with a `[gateway]` table, keeping the previous file as `.bak`; `--plan` shows both. It then issues the gateway's certificate from the cluster CA, initializes the gateway ledger, and ensures the `scribe-gateway` service account with namespace reader and delegate grants in every namespace it fronts and its credential. Repositories created directly on a store before the gateway existed are imported by its first inventory poll and keep their IDs; through the gateway their names carry the namespace (`studio/game`). The cluster must be stopped. `up` then starts auth, then storage, then the gateway, and `down` drains them in reverse. A client logs in once more with the gateway as `--server`: a saved session belongs to the server it was made for. On a storage-only cluster `add gateway` is refused (run `add auth` first), as it is when a store listens on port 0 (the gateway needs its fixed address). On a cluster that already has a gateway it changes nothing and says `nothing to add`. Resume and refusal work as for [add auth](/docs/reference/cli/scribe-ctl.md#add-auth). ## upgrade Upgrade the cluster to the release in a directory of executables: sanity checks, backups of the roots a migration changes, explicit migrations, a health check, and rollback on failure. Resumable; one at a time ```text scribe-ctl upgrade [OPTIONS] --binaries scribe-ctl upgrade [OPTIONS] --source ``` | Argument or option | Meaning | | --- | --- | | `--binaries ` | Native: directory with the new release's executables: scribe-ctl and those the cluster runs (scribe-auth, scribe-server, scribe-gateway); a scribe client beside them is reported. A newer scribe-ctl there takes the upgrade over | | `--source ` | Docker: the Scribe checkout of this scribe-ctl's release to build the new images from (and the web interface's, with [web]); built before planning, also with --plan | | `--plan` | Show the plan and run every read-only check and migration dry run, without writing anything, taking a lock or touching a service | `upgrade` moves a set-up cluster from its active set to the release in `DIR`: natively a directory of executables (`--binaries`), under docker a Scribe checkout whose images it builds first (`--source`, see [Docker runtime](/docs/reference/cli/scribe-ctl.md#docker-runtime)); the other option is refused. It needs a recorded set (see [setup](/docs/reference/cli/scribe-ctl.md#setup)): a cluster without `bin/current` is refused with a pointer to `setup`. **Which scribe-ctl runs it.** Only a release's own controller knows that release's migrations. Under docker the images are built as the running `scribe-ctl`'s own release, so there is nothing to hand over: build `scribe-ctl` from the checkout and run it. When `DIR` holds a `scribe-ctl` of a newer version, or of the same version and another build, the running `scribe-ctl` hands the whole command over to it: it runs `DIR/scribe-ctl` with the same options (`--plan` and `--json` included) and exits with its exit code. An older `scribe-ctl` in `DIR` is not handed anything; the plan refuses it as a downgrade. The plan's `controller` check names who planned it. **Checks.** Every check below runs for real under `--plan` too, and any failure refuses the whole upgrade before anything changes: - `new set`: `DIR` holds `scribe-ctl` and every executable the cluster runs; each answers `--build-info` (see [JSON output](/docs/json-output.md#build-info)), names itself correctly and is built for this machine's OS and architecture; all report one version and build. A `scribe` client beside them is reported, never required or installed. - `release`: the new version is not older than the active one (SemVer order); the same version with another build is allowed. Upgrading to the active build changes nothing and says so. - `formats`: which persistent formats and protocols change between the sets, from the executables' build info. A format going back refuses the upgrade. - ` data`: each data root's recorded formats, read without opening or locking it (the storage data-root descriptor, the `SCRIBE_AUTH` marker, the `SCRIBE_GATEWAY` marker), against what the new executables read and write. Data newer than they support is refused. - `permissions`: existing secret files, and auth's data root, grant nothing to other accounts. Auth refuses to start (or migrate) from a root others can enter, so a mode like 0755 fails the plan with the `chmod 700 ` that fixes it, before anything is drained. - `cluster`, `journal`: a busy cluster, or another command's interrupted run, is refused. **Migrations.** In dependency order, each after a backup: auth with the new `scribe-auth migrate` (which rewrites the revision witness), each store with the new `scribe-server migrate`, and under docker the web database with the new worker image's `migrate.js`. While the cluster is stopped, the plan runs `scribe-auth migrate --dry-run` and `scribe-server migrate --dry-run` with the new executables and migrates exactly the roots they report pending (a refused dry run refuses the upgrade). While it runs, a root's data-root lock rules a dry run out, so a root is migrated when its formats change between the sets or it records an older format than the new executable; each `migrate` checks everything before it writes. Two migrations are refused with what to do by hand: auth's schema 3 to 4, which replaces every credential (the administrator's, every server's and service account's), and any of the gateway ledger, since `scribe-gateway` has no migrate command. The web database migrates with every web release, since `migrate.js` records the release's schema even when no migration is pending. While the old release serves, the plan runs the new worker's `migrate.js --dry-run` against the running database, so data the new release would refuse stops the upgrade before anything stops; a stopped database is dry-run by the migrate step itself, first. **Auth's revision witness.** Auth refuses to start on a database behind its witness, the file that detects a restored old copy. The witness stays outside auth's data root and its backups, so the upgrade records it (a line of incarnation and revision) in its journal when it backs auth up, after the drain, before copying. A rollback that restores auth writes the recorded witness back with the database, so the two agree again; the journal, and with it the only copy, is removed when the upgrade completes, so no backup ever carries a witness that would let an old copy pass. A change made through the new release's auth between its start and a rollback is undone with the rest. **Steps.** Install the new set side by side in `/bin//` (each executable copied, checked to answer as its source did, then `SET.json`; under docker the images recorded), while the cluster still serves; drain it if it runs; back up each root that will be migrated; switch `bin/current`; migrate; start every service from the new set and wait until each is ready (the health check, as `up --daemon`); stop again if the cluster was stopped before; rewrite `CLUSTER-INFO.txt`. For a running gateway cluster with nothing to migrate: ```text $ scribe-ctl --config ./scribe-cluster.toml upgrade --binaries releases/next --plan Handing the upgrade over to /srv/scribe/releases/next/scribe-ctl of 0.1.0 (build next), which knows its release's migrations… Scribe · local · upgrade plan · gateway variant · native runtime Definition /srv/scribe/scribe-cluster.toml Paths below are relative to its directory. ENDPOINTS auth tls://127.0.0.1:7443 storage-a tls://127.0.0.1:7447 gateway tls://127.0.0.1:7557 CHECKS ok release 0.1.0 → 0.1.0 (build next): the same version, another build ok installed set 0.1.0 in cluster-state/local/bin/0.1.0 ok new set releases/next: scribe-ctl, scribe-auth, scribe-server, scribe-gateway 0.1.0 (build next) for macos/aarch64, and the scribe client (not installed) ok controller scribe-ctl 0.1.0 (build next) from releases/next (handed over by scribe-ctl 0.1.0) ok formats unchanged: every executable reads and writes what the active set does ok cluster running; drained after the new set is installed ok permissions 10 existing secret files and auth's data root are private ok auth data auth_schema 4; no migration: its formats do not change (no dry run: auth serves it and holds its data-root lock) ok storage-a data catalog_schema 1, envelope 1, object_store 1, repository_schema 3; no migration: its formats do not change (no dry run: storage-a serves it and holds its data-root lock) ok gateway data route_store_schema 2; no migration STEPS 1. Install scribe-ctl, scribe-auth, scribe-server, scribe-gateway 0.1.0 (build next) side by side in cluster-state/local/bin/next (from releases/next) side by side with the active set, before anything stops 2. Drain the running cluster roots are backed up, migrated and switched only while nothing holds them 3. Switch the active set from 0.1.0 to next (bin/current) services start from the set bin/current names 4. Start the cluster with next and wait until every service is ready the health check: every service must become ready 5. Write the cluster summary CLUSTER-INFO.txt the summary names the active set ROLLBACK (if step 2 or a later one fails) 6. Drain whatever of the cluster runs nothing may hold a root while it is restored 7. Switch the active set from next to 0.1.0 (bin/current) back to the executables that served before 8. Start the cluster again with 0.1.0 it was running before the upgrade 9. Write the cluster summary CLUSTER-INFO.txt the summary names the active set NOTES - The scribe-ctl in releases/next decides this upgrade: a release's own scribe-ctl runs its upgrade, because only it knows that release's migrations. An older scribe-ctl given a newer one in --binaries hands the upgrade over to it. - No data root changes, so nothing is backed up. - If step 2 or a later one fails, the rollback below runs and the cluster returns to 0.1.0. - A failure before step 2 changes nothing: 0.1.0 stays active and the cluster keeps serving; run upgrade again to continue. - Interrupted (a crash, a power loss)? Run the same command again: it resumes from cluster-state/local/journal.json, or finishes a rollback. AFTERWARDS - Run cluster-state/local/bin/next/scribe-ctl from now on (or copy it where you keep scribe-ctl): it is this release's controller. Plan only: nothing was written and no lock was taken. Run upgrade without --plan to apply it. ``` For the same cluster, stopped, with a pending migration of its store: ```text $ scribe-ctl --config ./scribe-cluster.toml upgrade --binaries releases/next --plan Handing the upgrade over to /srv/scribe/releases/next/scribe-ctl of 0.1.0 (build next), which knows its release's migrations… … CHECKS ok release 0.1.0 → 0.1.0 (build next): the same version, another build ok installed set 0.1.0 in cluster-state/local/bin/0.1.0 ok new set releases/next: scribe-ctl, scribe-auth, scribe-server, scribe-gateway 0.1.0 (build next) for macos/aarch64, and the scribe client (not installed) ok controller scribe-ctl 0.1.0 (build next) from releases/next (handed over by scribe-ctl 0.1.0) ok formats unchanged: every executable reads and writes what the active set does ok cluster stopped; started for the health check, then stopped again ok permissions 10 existing secret files and auth's data root are private ok auth data auth_schema 4; dry run: nothing to migrate ok storage-a data catalog_schema 1, envelope 1, object_store 1, repository_schema 2; dry run: the data-root descriptor is upgraded; backed up, then migrated ok gateway data route_store_schema 2; no migration ok free space 177.33 KiB to copy to cluster-state/local/backups/20260927T103411654Z, 424.11 GiB available STEPS 1. Install scribe-ctl, scribe-auth, scribe-server, scribe-gateway 0.1.0 (build next) side by side in cluster-state/local/bin/next (from releases/next) side by side with the active set, before anything stops 2. Back up storage storage-a (data/storage-a) to cluster-state/local/backups/20260927T103411654Z/storage-a the migration changes it; a rollback restores this copy 3. Switch the active set from 0.1.0 to next (bin/current) services start from the set bin/current names 4. Migrate storage storage-a (data/storage-a) with the new scribe-server migrate explicit, with the new executables; reruns resume where it stopped 5. Start the cluster with next and wait until every service is ready the health check: every service must become ready 6. Stop the cluster again, as it was before it was stopped before the upgrade 7. Write the cluster summary CLUSTER-INFO.txt the summary names the active set ROLLBACK (if step 3 or a later one fails) 8. Drain whatever of the cluster runs nothing may hold a root while it is restored 9. Restore storage storage-a from cluster-state/local/backups/20260927T103411654Z/storage-a if it was migrated, keeping the migrated root as data/storage-a.upgraded-20260927T103411654Z the previous release cannot read what the migration wrote 10. Switch the active set from next to 0.1.0 (bin/current) back to the executables that served before 11. Write the cluster summary CLUSTER-INFO.txt the summary names the active set … ``` **Backups.** Each root a migration will change (auth, a store, the web database) is copied, stopped, to `/backups///` (`auth`, the store's name, `web`): directories private (0700), files keeping their modes, every file and directory flushed, and complete only once `.json` (its manifest) is written. The manifest lists every directory and every file with its size and BLAKE3 digest; the copy is read back and checked against them before the manifest is written. The web database backup is a cold copy of its SurrealDB files, consistent because the database is stopped. Roots that do not change are not copied. The plan's `free space` check measures the roots to copy and refuses the upgrade unless the filesystem holding `/backups` has room for them plus a reserve of a tenth of their size, at least 1 GiB (a full filesystem would stop the services on it too). Backups and earlier binary sets are kept; `scribe-ctl` never deletes either. `where` lists them. **Rollback.** A failure once a running cluster is drained (for a stopped cluster, from the switch on) rolls back: drain whatever runs, restore each root whose migration started (auth with its witness; the migrated root is moved aside to `.upgraded-` and kept; the backup is checked against its manifest, copied beside it, checked again and renamed into place), switch `bin/current` back, start the previous set if the cluster ran before, and report. So a backup that fails after the drain restarts the old set rather than leaving the cluster down. The new set stays installed. A failure earlier (the install) changes nothing and leaves the previous set serving. For example, a release whose storage server cannot start: ```text $ scribe-ctl --config ./scribe-cluster.toml upgrade --binaries releases/next Handing the upgrade over to /srv/scribe/releases/next/scribe-ctl of 0.1.0 (build next), which knows its release's migrations… local · [1/5] Install scribe-ctl, scribe-auth, scribe-server, scribe-gateway 0.1.0 (build next) side by side in cluster-state/local/bin/next (from releases/next)… local · [2/5] Drain the running cluster… local · [3/5] Switch the active set from 0.1.0 to next (bin/current)… local · [4/5] Start the cluster with next and wait until every service is ready… local · [6/9] Drain whatever of the cluster runs… local · [7/9] Switch the active set from next to 0.1.0 (bin/current)… local · [8/9] Start the cluster again with 0.1.0… local · [9/9] Write the cluster summary CLUSTER-INFO.txt… Scribe · local · Upgrade to next rolled back step 4 (Start the cluster with next and wait until every service is ready) failed: supervisor exited (exit status: 1): storage-a failed to start; inspect /srv/scribe/cluster-state/local/logs/storage-a.log; its log is /srv/scribe/cluster-state/local/logs/supervisor.log installed 0.1.0 (build next) in cluster-state/local/bin/next drained the cluster switched the active set from 0.1.0 to next ROLLED BACK nothing was running switched the active set from next to 0.1.0 every service is ready, running 0.1.0 wrote CLUSTER-INFO.txt The active set is 0.1.0 again, and the cluster runs. The new set stays installed; fix the cause, then run upgrade again. ``` **Docker.** The same plan, with images: for a running storage cluster with the web interface, upgraded with the next release's `scribe-ctl` (build `next`): ```text $ scribe-ctl --config ./scribe-cluster.toml upgrade --source /srv/scribe-src --plan local · Building scribe/auth:next from /srv/scribe-src (the first build takes a while)… local · Building scribe/storage:next from /srv/scribe-src (the first build takes a while)… local · Building scribe/gateway:next from /srv/scribe-src (the first build takes a while)… local · Building the web interface's images (api, …) from /srv/scribe-src/apps/web… Scribe · local · upgrade plan · storage variant · docker runtime Definition /srv/scribe/scribe-cluster.toml Paths below are relative to its directory. ENDPOINTS storage-a tcp://127.0.0.1:7447 web http://localhost:8080 CHECKS ok release 0.1.0 → 0.1.0 (build next): the same version, another build ok installed set 0.1.0 in cluster-state/local/bin/0.1.0 ok new set /srv/scribe-src: scribe-auth, scribe-server, scribe-gateway 0.1.0 (build next) for linux/x86_64 ok controller scribe-ctl 0.1.0 (build next) ok formats unchanged: every executable reads and writes what the active set does ok cluster running; drained after the new set is installed … ok web data dry run: no migration pending; data/web is backed up, then the new worker's migrate.js runs (its dry run first) … STEPS 1. Record the images scribe/auth:next, scribe/storage:next, scribe/gateway:next and the web interface's of 0.1.0 (build next) as a set in cluster-state/local/bin/next side by side with the active set, before anything stops 2. Drain the running cluster roots are backed up, migrated and switched only while nothing holds them 3. Back up the web database (data/web) to cluster-state/local/backups/20260927T112353229Z/web the migration changes it; a rollback restores this copy 4. Switch the active set from 0.1.0 to next (bin/current) services start from the set bin/current names 5. Migrate the web database (data/web) with the new worker's migrate.js, after its dry run explicit, with the new executables; reruns resume where it stopped 6. Start the cluster with next and wait until every service is ready the health check: every service must become ready 7. Write the cluster summary CLUSTER-INFO.txt the summary names the active set ROLLBACK (if step 2 or a later one fails) 8. Drain whatever of the cluster runs nothing may hold a root while it is restored 9. Restore the web database from cluster-state/local/backups/20260927T112353229Z/web if it was migrated, keeping the migrated root as data/web.upgraded-20260927T112353229Z the previous release cannot read what the migration wrote 10. Switch the active set from next to 0.1.0 (bin/current) back to the executables that served before 11. Start the cluster again with 0.1.0 it was running before the upgrade 12. Write the cluster summary CLUSTER-INFO.txt the summary names the active set NOTES - This scribe-ctl decides this upgrade: it built the images from /srv/scribe-src as its own release, because only a release's own scribe-ctl knows its migrations. - Backups go to cluster-state/local/backups/20260927T112353229Z; they and the previous set (cluster-state/local/bin/0.1.0) are kept. scribe-ctl never deletes either; remove them yourself once the new release has proven itself. - If step 2 or a later one fails, the rollback below runs and the cluster returns to 0.1.0 with its data as backed up. - A failure before step 2 changes nothing: 0.1.0 stays active and the cluster keeps serving; run upgrade again to continue. - Interrupted (a crash, a power loss)? Run the same command again: it resumes from cluster-state/local/journal.json, or finishes a rollback. AFTERWARDS - Keep running this scribe-ctl (built from the same checkout as the images): it is this release's controller. Plan only: nothing was written and no lock was taken. Run upgrade without --plan to apply it. ``` **Resume and lock.** The upgrade records its decisions (sets, backup directory, stores to migrate, whether the cluster ran) and every step in the journal before it runs. After a crash or power loss, run the same command again: it continues from the journal, moving a partially copied set or backup aside as `.interrupted-`. A rollback that stopped is finished by the next run from the journal alone. The upgrade holds the command lock for its whole run, so a second upgrade, `up`, `down`, `setup` or `add` is refused meanwhile; `--plan` takes no lock. Exit 0 means upgraded (or nothing to upgrade), 1 means refused, failed or rolled back. With `--json`, a completed upgrade reports `plan` and `results`; a rolled back one reports `ok: false`, `rolled_back: true`, `failure`, `results` (the steps before the failure), `rollback`, `active_set` and `backups`. ## prune Remove what old releases left behind, and only what scribe-ctl made: binary sets other than the active one and the --keep newest before it (for a rollback), the images tagged with those sets' builds that no container uses, the scribe-ctl builds a remote command installed for them, and upgrade backups older than the --keep newest. Docker's build cache, other images, volumes and containers are never touched. Never run by renew; you run it when you choose ```text scribe-ctl prune [OPTIONS] ``` | Argument or option | Meaning | | --- | --- | | `--plan` | Show what would be removed, and why each is old, without removing anything or taking a lock | | `--keep ` | How many sets before the active one stay, and how many of the newest upgrade backups [default: 1] | Sets are ordered by when they were installed. An image a container uses (running or stopped) stays, and the plan names it. A directory in the cluster's `bin/` counts as a scribe-ctl build only when its name is a build id and it holds `scribe-ctl`; anything else there is left alone. The removals are independent and each is idempotent, so an interrupted prune is simply run again. Exit 0 when pruned or nothing is old, 1 when refused (no active set is recorded). ## renew Replace everything the cluster renews that expires within 30 days: service credentials and the administrator credential (issued, put in place, verified, then the old one revoked) and server certificates (reissued for the key they have by the CA that signed them, then a TLS handshake proves each); and retire the old CA of every rotation whose overlap has ended. Holders restart one at a time (or a stopped cluster starts); a failed verification puts the old files back. The web interface's own certificate stays yours. Non-interactive, for schedulers: exit 0 when renewed or nothing is due, 1 with the old files kept. With --rotate-ca it starts a CA rotation instead (never done by a plain renew); with --rekey it replaces servers' private keys instead. Resumable; one at a time ```text scribe-ctl renew [OPTIONS] ``` | Argument or option | Meaning | | --- | --- | | `--plan` | Show what would be renewed, rotated or retired and when each expires, running every read-only check, without writing anything, taking a lock or touching a service | | `--ca-dir ` | The storage and gateway CA's directory (its ca.pem and ca-key.pem) when its key is not kept beside the cluster. With --remote the key stays on this machine: certificates are signed here and only they travel to the cluster | | `--auth-ca-dir ` | The auth CA's directory (its ca.pem and ca-key.pem) when its key is not kept beside the cluster; as --ca-dir | | `--rotate-ca ` | Start a CA rotation (a plain renew never does). storage or auth, CAs clients trust: a new CA and a cross certificate the old one signs for it; every server certificate of that CA is reissued by the new CA for its key and followed by the cross certificate, so clients that trust only the old CA keep connecting and learn the new one during the overlap, after which a plain renew retires the old CA. credential or access, auth's internal client CAs (only Scribe's services trust them): a successor is added while the cluster drains for a moment | | `--retire-ca ` | Finish a rotation now. storage or auth: before its overlap ends (a plain renew retires it at the end), so clients that have not connected since it started will see "CA changed" and trust the new CA by hand. credential or access: retire auth's oldest internal CA, which auth refuses while anything it signed is in use | | `--rekey [...]` | Replace TLS servers' private keys (a plain renew never does): each named service (comma-separated or repeated; every TLS server of the cluster when none is named) gets a new key, generated where it runs, and a certificate for it from the CA that signs it now, for the same names. The old key and certificate are kept until a TLS handshake proves the new pair, and go back if it fails. Clients trust the CA, not the key: nothing changes for them. For a key suspected leaked, or to move servers to a post-quantum key | | `--overlap-days ` | With --rotate-ca storage\|auth: the days both CAs are trusted, which is how long clients have to connect and learn the new CA [default: 90] | `` is `storage` (the storage and gateway CA, `certificates.ca_dir`), `auth` (the auth CA, `certificates.auth_ca_dir`), `credential` or `access` (auth's internal client CAs). ### What renew owns - **Credentials**: each authenticated store's server credential, the gateway's (`scribe-gateway`), the `scribe-monitor` inventory credential, under docker with the web interface the `scribe-web` reader's, and the administrator credential the definition names (`[auth] admin_credential`; the bootstrap one setup writes never expires, an operator's does). Each file's expiry is read from the file itself; one within 30 days, or expired, is due. - **Server certificates** of auth, every store and the gateway, due the same way. The web interface's `fullchain.pem` is yours ([web interface](/docs/reference/cli/scribe-ctl.md#web-interface)). - **CA rotations** whose overlap has ended: their old CA is retired ([CA rotation](/docs/reference/cli/scribe-ctl.md#ca-rotation)). With nothing due, renew exits 0 and writes nothing, not even a lock: ```text $ scribe-ctl --config ./scribe-cluster.toml renew Scribe · local · Unchanged: nothing to renew: 4 credentials are valid for 30 more days; the first to expire is the storage-a server credential on 2027-09-27 3 server certificates are valid for 30 more days; the first to expire is auth's on 2027-09-27 ``` ### Credentials For the due credentials it runs, in order: 1. auth issues each replacement into `.next` (365 days; an administrator credential 3650) for the same holder; the old one stays valid; 2. each is put in place: the current file is kept as `.previous` (durably), then the new one is renamed over it; 3. the services that read their credential only when they start restart: the stores, then the gateway, one at a time while the others keep serving (natively the supervisor stops and starts each in place and waits until it is ready; under docker its container is recreated). A service that does not come back stops the cluster, as one that exits does, and the renewal rolls back. The inventory, web reader and administrator credentials are read on every use and need no restart. A stopped cluster is started instead, for the proof, and stopped again at the end; 4. each new credential is proved to work for its purpose: a store serves a namespace read (the inventory, read as `scribe-monitor` or else the gateway's account), which it authorizes through auth with its own new credential; the gateway reports a poll of every instance made after the proof began; the inventory and web reader credentials each read every store's namespace themselves (the web reader through auth's published address, since its file names auth as the containers reach it); the administrator credential answers an administrator request; 5. only then is each replaced credential revoked and its kept copy removed. ```text $ scribe-ctl --config ./scribe-cluster.toml renew --plan Scribe · local · renew plan · gateway variant · native runtime Definition /srv/scribe/scribe-cluster.toml Paths below are relative to its directory. ENDPOINTS auth tls://127.0.0.1:7443 storage-a tls://127.0.0.1:7447 gateway tls://127.0.0.1:7557 CHECKS ok credentials 2 of 4 credentials expire within 30 days: the storage-a server credential expires on 2026-09-28; the gateway credential (scribe-gateway) expires on 2026-09-28 ok certificates 3 server certificates are valid for 30 more days; the first to expire is auth's on 2027-09-27 ok cluster running; storage-a, gateway restart to read the renewed credentials and certificates, one at a time while the rest serve ok permissions 10 existing secret files and auth's data root are private STEPS 1. Issue a new storage-a server credential for 365 days into secrets/storage-a.credential.next it expires on 2026-09-28; the old one stays valid until it is revoked 2. Issue a new gateway credential (scribe-gateway) for 365 days into secrets/gateway.credential.next it expires on 2026-09-28; the old one stays valid until it is revoked 3. Put the new storage-a server credential in place as secrets/storage-a.credential, keeping the current one as secrets/storage-a.credential.previous the holder reads this file; a rollback puts the kept copy back 4. Put the new gateway credential (scribe-gateway) in place as secrets/gateway.credential, keeping the current one as secrets/gateway.credential.previous the holder reads this file; a rollback puts the kept copy back 5. Restart storage-a, gateway so they read their renewed files they read these files only when they start; the supervisor restarts each in turn while the others serve 6. Verify the new storage-a server credential: storage-a serves a namespace read, which it authorizes with it the old credential is revoked only once the new one is proved to work 7. Verify the new gateway credential (scribe-gateway): the gateway polls every instance with it the old credential is revoked only once the new one is proved to work 8. Revoke the replaced storage-a server credential and remove its kept copy secrets/storage-a.credential.previous nothing uses it any more 9. Revoke the replaced gateway credential (scribe-gateway) and remove its kept copy secrets/gateway.credential.previous nothing uses it any more ROLLBACK (if one of steps 3-7 fails) 10. Put the previous gateway credential (scribe-gateway) back from secrets/gateway.credential.previous into secrets/gateway.credential the old credential is still valid 11. Put the previous storage-a server credential back from secrets/storage-a.credential.previous into secrets/storage-a.credential the old credential is still valid 12. Restart storage-a, gateway so they read the previous files again back to the files that served before 13. Revoke the new storage-a server credential this renewal issued an unproved credential is not left valid 14. Revoke the new gateway credential (scribe-gateway) this renewal issued an unproved credential is not left valid NOTES - Nothing old stops working before its replacement is proved; if a verification fails, the old files go back in place and the new credentials are revoked. Plan only: nothing was written and no lock was taken. Run renew without --plan to apply it. $ scribe-ctl --config ./scribe-cluster.toml renew local · [1/9] Issue a new storage-a server credential for 365 days into secrets/storage-a.credential.next… … local · [9/9] Revoke the replaced gateway credential (scribe-gateway) and remove its kept copy secrets/gateway.credential.previous… Scribe · local · Renewal complete storage-a: issued a new credential for 365 days into secrets/storage-a.credential.next scribe-gateway: issued a new credential for 365 days into secrets/gateway.credential.next put the new credential in place as secrets/storage-a.credential; the current one is kept as secrets/storage-a.credential.previous until it is revoked put the new credential in place as secrets/gateway.credential; the current one is kept as secrets/gateway.credential.previous until it is revoked restarted storage-a, gateway, which read their renewed files storage-a: served a namespace read (0 repositories), authorized with the new credential gateway: polled every instance with the new credential storage-a: revoked the replaced credential dfbb4e5c3a71 scribe-gateway: revoked the replaced credential 3eaa5e0b51fa RENEWED storage-a server credential: secrets/storage-a.credential, valid until 2027-09-27 gateway credential (scribe-gateway): secrets/gateway.credential, valid until 2027-09-27 ``` If a step from the first put-in-place through the last proof fails, renew rolls back: the kept credentials go back in place, the holders restart with them (or the cluster is stopped again), and the new credentials are revoked. It exits 1; the old credentials stay valid until their own expiry, and the next run tries again: ```text $ scribe-ctl --config ./scribe-cluster.toml renew local · [1/4] Issue a new inventory credential (scribe-monitor) for 365 days into secrets/monitor.credential.next… local · [2/4] Put the new inventory credential (scribe-monitor) in place as secrets/monitor.credential, keeping the current one as secrets/monitor.credential.previous… local · [3/4] Verify the new inventory credential (scribe-monitor): an inventory read with it… local · [5/6] Put the previous inventory credential (scribe-monitor) back from secrets/monitor.credential.previous into secrets/monitor.credential… local · [6/6] Revoke the new inventory credential (scribe-monitor) this renewal issued… Scribe · local · Renewal rolled back step 3 (Verify the new inventory credential (scribe-monitor): an inventory read with it) failed: … scribe-monitor: issued a new credential for 365 days into secrets/monitor.credential.next put the new credential in place as secrets/monitor.credential; the current one is kept as secrets/monitor.credential.previous until it is revoked ROLLED BACK put the previous inventory credential (scribe-monitor) back into secrets/monitor.credential scribe-monitor: revoked the new credential aa64db7532cf The previous inventory credential (scribe-monitor) is in place again, valid until 2026-09-28. Fix the cause, then run renew again. ``` A failure before anything is put in place (auth unreachable while issuing) leaves the old credentials in place and the issued ones in `.next`; a failure while revoking leaves the proved new credentials in place and the old ones still valid. Either exits 1, and the next run resumes where it stopped. ### Server certificates A due certificate is reissued for the key it already names (the server's private key is never read or replaced), for the names the definition gives the service, by the CA that signed it, for 365 days (never past the CA itself; a CA with less than 30 days left is refused: rotate it first). The file holds the certificate followed by its CA, so what the server presents ends with the CA clients trust and fingerprint. The file is put in place like a credential (kept as `.previous`), only that service restarts, and a TLS handshake trusting the CA alone, as a client does, proves it before the kept copy is removed; clients notice nothing: ```text $ scribe-ctl --config ./scribe-cluster.toml renew --plan Scribe · local · renew plan · gateway variant · native runtime Definition /srv/scribe/scribe-cluster.toml Paths below are relative to its directory. ENDPOINTS auth tls://127.0.0.1:7443 storage-a tls://127.0.0.1:7447 gateway tls://127.0.0.1:7557 CHECKS ok credentials nothing to renew: 4 credentials are valid for 30 more days; the first to expire is the storage-a server credential on 2027-09-27 ok certificates 1 of 3 server certificates expire within 30 days: storage-a's expires on 2026-10-09; each is reissued for the key it has ok cluster running; storage-a restarts to read the renewed credentials and certificates, one at a time while the rest serve ok permissions 10 existing secret files and auth's data root are private STEPS 1. Reissue storage-a's server certificate for its key into pki/storage-a/server.pem.next, signed with the CA key in pki/ca, followed by its CA it expires on 2026-10-09; the same key, the same CA 2. Put storage-a's new certificate in place as pki/storage-a/server.pem, keeping the current one as pki/storage-a/server.pem.previous the server reads this file when it starts; a rollback puts the kept copy back 3. Restart storage-a so it reads its renewed files they read these files only when they start; the supervisor restarts each in turn while the others serve 4. Verify a TLS handshake with storage-a trusting the current storage and gateway CA alone clients hold one CA each; every one of them must still connect 5. Remove storage-a's replaced certificate pki/storage-a/server.pem.previous the new certificate is proved ROLLBACK (if one of steps 2-4 fails) 6. Put storage-a's previous certificate back from pki/storage-a/server.pem.previous into pki/storage-a/server.pem the old certificate is still valid 7. Restart storage-a so it reads the previous files again back to the files that served before NOTES - Nothing old stops working before its replacement is proved; if a verification fails, the old files go back in place and the new credentials are revoked. Plan only: nothing was written and no lock was taken. Run renew without --plan to apply it. $ scribe-ctl --config ./scribe-cluster.toml renew local · [1/5] Reissue storage-a's server certificate for its key into pki/storage-a/server.pem.next, signed with the CA key in pki/ca, followed by its CA… … local · [5/5] Remove storage-a's replaced certificate pki/storage-a/server.pem.previous… Scribe · local · Renewal complete issued a new certificate for pki/storage-a/server.pem's key into pki/storage-a/server.pem.next, valid for 365 days put the new certificate in place as pki/storage-a/server.pem; the current one is kept as pki/storage-a/server.pem.previous restarted storage-a, which reads its renewed files storage-a: TLS handshake at 127.0.0.1:7447 as localhost verified trusting the current storage and gateway CA alone storage-a: removed the replaced certificate pki/storage-a/server.pem.previous RENEWED storage-a's certificate: pki/storage-a/server.pem, valid until 2027-09-27 ``` The CA's key signs where renew runs. Setup keeps it beside the cluster (`ca-key.pem` in the CA directory); moved elsewhere, name its directory with `--ca-dir DIR` (the storage and gateway CA) or `--auth-ca-dir DIR` (auth's). Without it the plan fails for exactly that certificate and names the command: ```text $ scribe-ctl --config ./scribe-cluster.toml renew --plan … CHECKS ok credentials nothing to renew: 4 credentials are valid for 30 more days; the first to expire is the storage-a server credential on 2027-09-27 ok certificates 1 of 3 server certificates expire within 30 days: storage-a's expires on 2026-10-09; each is reissued for the key it has ok cluster running; storage-a restarts to read the renewed credentials and certificates, one at a time while the rest serve ok permissions 9 existing secret files and auth's data root are private FAIL ca keys the storage and gateway CA's key is not beside the cluster (pki/ca/ca-key.pem is missing): run `scribe-ctl --config /srv/scribe/scribe-cluster.toml renew --ca-dir DIR` on the machine that holds it, where DIR holds its ca.pem and ca-key.pem (with --remote from there, only the new certificates travel to the cluster) … ``` With `--remote`, DIR is on the machine you run scribe-ctl on: the remote scribe-ctl says what needs signing (each certificate's public part and names), this machine signs with the key in DIR and pushes only the certificates (`upload/renew/signed.json` in the cluster's directory), and the remote checks each (the same key, issued by its CA, exactly its names) before it puts it in place. The key never leaves this machine. `--plan` with `--ca-dir` shows the remote plan as if this machine signs. ### Server keys A plain renew keeps each server's private key. `--rekey [SERVICE,...]` replaces it, for a key suspected leaked or to move a server to a post-quantum (ML-DSA-65) key: for each named TLS server (every one when none is named) a new key is generated beside the current one (`.next`, owner-only, written atomically), the CA that signs the server now certifies it for the same names (followed by the cross certificate during a rotation's overlap, as a reissue is), key and certificate are put in place together (the old ones kept as `.previous`), only that service restarts, and a TLS handshake trusting the CA clients hold proves the new pair before the kept key and certificate are removed. A failure from the key's put-in-place through the handshake puts the old key and certificate back together. Clients trust the CA, not the key, so none of them changes anything. A rekey run does only the rekey; nothing that expires is renewed alongside. ```text $ scribe-ctl --config ./scribe-cluster.toml renew --rekey storage-a local · [1/8] Generate a new private key for storage-a into pki/storage-a/server-key.pem.next (only this account can read it)… local · [2/8] Issue storage-a's server certificate for its new key pki/storage-a/server-key.pem.next into pki/storage-a/server.pem.next, signed with the CA key in pki/ca, followed by its CA… local · [3/8] Put storage-a's new private key in place as pki/storage-a/server-key.pem, keeping the current one as pki/storage-a/server-key.pem.previous… local · [4/8] Put storage-a's new certificate in place as pki/storage-a/server.pem, keeping the current one as pki/storage-a/server.pem.previous… local · [5/8] Restart storage-a so it reads its renewed files… local · [6/8] Verify a TLS handshake with storage-a trusting the current storage and gateway CA alone… local · [7/8] Remove storage-a's replaced certificate pki/storage-a/server.pem.previous… local · [8/8] Remove storage-a's replaced private key pki/storage-a/server-key.pem.previous… Scribe · local · Renewal complete … RENEWED storage-a's certificate: pki/storage-a/server.pem, valid until 2027-09-28, for a new key ``` A name that is not a TLS server of the cluster fails the plan (`nope is not a TLS server of this cluster; its TLS servers are auth, storage-a`). With `--remote` and `--ca-dir DIR`, the remote scribe-ctl stages the new key there first and asks for its public key to be certified (`public_key` in its request); this machine certifies it with the key in DIR, and only the certificate travels back. The new private key never leaves the cluster's machine. A `.next` a request staged and no run used is what the next `--rekey` of that service certifies and puts in place; `doctor` reports it (and any `.next` or `.previous` beside a certificate or key with no renewal pending) as `renew.leftovers.`. ### CA rotation `--rotate-ca storage|auth` replaces a CA through a signed handover, never started by a plain renew (which only warns, from a year before the CA expires). It creates a new CA and a *cross certificate*: the new CA's name and key, a CA, issued by the old CA, valid for the overlap (90 days, or `--overlap-days`, never past the old CA). Every server certificate of that CA is reissued in the same run by the new CA for the key it has, followed by the cross certificate, so a client that trusts only the old CA verifies it through the cross certificate (and learns the new CA), and a client given the new CA verifies it directly. Before any server changes, the web interface's services that read `/etc/scribe/trust` restart trusting both CAs (Node follows no cross certificate: it refuses a chain through one to an old CA that may sign no intermediate). The servers restart, handshakes trusting the old and the new CA alone prove them, then the new CA becomes `ca.pem` in the CA directory (the old one and its key move to `previous/`, the cross certificate to `cross.pem`), the trust files the definition names beside it (a store's `ca`, gateway instances' `ca`, auth's `ca`, a `ca.pem` beside a server certificate) hold both CAs, for the auth CA every credential file ctl manages is rewritten to carry the new auth CA, and the services that read the new trust restart (the gateway for the storage CA; the stores and the gateway for auth's; the web services). The rotation is recorded in the cluster's state directory (`ca-rotation.json`) with the day it retires. Any failure from the first certificate put in place through the last proof puts the old CA and certificates back and sets the new CA aside: ```text $ scribe-ctl --config ./scribe-cluster.toml renew --rotate-ca storage --plan Scribe · local · renew plan · gateway variant · native runtime Definition /srv/scribe/scribe-cluster.toml Paths below are relative to its directory. ENDPOINTS auth tls://127.0.0.1:7443 storage-a tls://127.0.0.1:7447 gateway tls://127.0.0.1:7557 CHECKS ok rotation starts a rotation of the storage and gateway CA: 2 server certificates (storage-a, gateway) move to a new CA together, followed by a cross certificate the old CA signs, so clients that trust only the old CA keep connecting for 90 days ok cluster running; storage-a, gateway restart to read the renewed credentials and certificates, one at a time while the rest serve ok permissions 10 existing secret files and auth's data root are private STEPS 1. Create the new storage and gateway CA in pki/ca/next and the cross certificate the old one signs for it, valid for the 90-day overlap the old CA signs the cross certificate that hands its trust to the new one 2. Reissue storage-a's server certificate for its key into pki/storage-a/server.pem.next, signed with the CA key in pki/ca/next, followed by the cross certificate every server of the CA moves to the new one in this rotation 3. Reissue gateway's server certificate for its key into pki/gateway/server.pem.next, signed with the CA key in pki/ca/next, followed by the cross certificate every server of the CA moves to the new one in this rotation 4. Put storage-a's new certificate in place as pki/storage-a/server.pem, keeping the current one as pki/storage-a/server.pem.previous the server reads this file when it starts; a rollback puts the kept copy back 5. Put gateway's new certificate in place as pki/gateway/server.pem, keeping the current one as pki/gateway/server.pem.previous the server reads this file when it starts; a rollback puts the kept copy back 6. Restart storage-a, gateway so they read their renewed files they read these files only when they start; the supervisor restarts each in turn while the others serve 7. Verify a TLS handshake with storage-a trusting the current storage and gateway CA, then the new storage and gateway CA alone clients hold one CA each; every one of them must still connect 8. Verify a TLS handshake with gateway trusting the current storage and gateway CA, then the new storage and gateway CA alone clients hold one CA each; every one of them must still connect 9. Make the new storage and gateway CA current in pki/ca (the old one moves to previous/) and record the rotation every server presents the new CA's certificates with the cross certificate 10. Restart gateway so it reads the new CA they read the CA only when they start 11. Verify a TLS handshake with storage-a trusting the current storage and gateway CA, then the old storage and gateway CA alone clients that learned the new CA and clients that know only the old one both connect 12. Verify a TLS handshake with gateway trusting the current storage and gateway CA, then the old storage and gateway CA alone clients that learned the new CA and clients that know only the old one both connect 13. Read through every store with a reader credential, and wait for the gateway's fresh inventory every store serves reads through the new trust 14. Remove storage-a's replaced certificate pki/storage-a/server.pem.previous the new certificate is proved 15. Remove gateway's replaced certificate pki/gateway/server.pem.previous the new certificate is proved ROLLBACK (if one of steps 4-13 fails) 16. Put gateway's previous certificate back from pki/gateway/server.pem.previous into pki/gateway/server.pem the old certificate is still valid 17. Put storage-a's previous certificate back from pki/storage-a/server.pem.previous into pki/storage-a/server.pem the old certificate is still valid 18. Make the old storage and gateway CA current again in pki/ca the old CA served before the rotation 19. Move the unused new storage and gateway CA aside in pki/ca a later rotation stages a CA of its own 20. Restart storage-a, gateway so they read the previous files again back to the files that served before NOTES - Nothing old stops working before its replacement is proved; if a verification fails, the old files go back in place and the new credentials are revoked. - After the rotation, clients learn the new storage and gateway CA whenever they connect during the 90-day overlap; `scribe-ctl --config /srv/scribe/scribe-cluster.toml status` shows the day a plain renew retires the old one. Clients that never connect in that time see "CA changed" and trust the new CA by hand. Plan only: nothing was written and no lock was taken. Run renew without --plan to apply it. $ scribe-ctl --config ./scribe-cluster.toml renew --rotate-ca storage local · [1/15] Create the new storage and gateway CA in pki/ca/next and the cross certificate the old one signs for it, valid for the 90-day overlap… … local · [15/15] Remove gateway's replaced certificate pki/gateway/server.pem.previous… Scribe · local · Renewal complete created the new storage and gateway CA and its cross certificate in pki/ca/next issued a new certificate for pki/storage-a/server.pem's key into pki/storage-a/server.pem.next, valid for 365 days issued a new certificate for pki/gateway/server.pem's key into pki/gateway/server.pem.next, valid for 365 days put the new certificate in place as pki/storage-a/server.pem; the current one is kept as pki/storage-a/server.pem.previous put the new certificate in place as pki/gateway/server.pem; the current one is kept as pki/gateway/server.pem.previous restarted storage-a, gateway, which read their renewed files storage-a: TLS handshake at 127.0.0.1:7447 as localhost verified trusting the current storage and gateway CA, and the new storage and gateway CA alone gateway: TLS handshake at 127.0.0.1:7557 as localhost verified trusting the current storage and gateway CA, and the new storage and gateway CA alone the new storage and gateway CA is current; the old one stays trusted through the cross certificate until 2026-12-26, when a plain renew retires it restarted gateway, which reads the new CA storage-a: TLS handshake at 127.0.0.1:7447 as localhost verified trusting the current storage and gateway CA, and the old storage and gateway CA alone gateway: TLS handshake at 127.0.0.1:7557 as localhost verified trusting the current storage and gateway CA, and the old storage and gateway CA alone storage-a: serves its repositories (0) from /srv/scribe/data/storage-a gateway: polled every instance storage-a: removed the replaced certificate pki/storage-a/server.pem.previous gateway: removed the replaced certificate pki/gateway/server.pem.previous RENEWED storage-a's certificate: pki/storage-a/server.pem, valid until 2027-09-27, from the new CA gateway's certificate: pki/gateway/server.pem, valid until 2027-09-27, from the new CA ROTATION The new storage and gateway CA is current: BLAKE3:a82af9cb26c76557c03c762ac30ff2c91473706a8f808574a7ed93859c2c67bc The old one (BLAKE3:8ca70e3c41276a59c247660fd4aa122117645038abc2e25ce8ba18560dd80f8c) stays trusted through the cross certificate for 90 days; a plain renew then retires it. Clients learn the new CA when they connect in that time; give new clients the new ca.pem. ``` `status` shows the rotation (and `doctor` its `ca.rotation.storage` check); a plain renew in the overlap changes nothing and says when it retires: ```text $ scribe-ctl --config ./scribe-cluster.toml status Scribe · local · running SERVICE STATE PID ENDPOINT auth running 46876 tls://127.0.0.1:7443 storage-a running 46907 tls://127.0.0.1:7447 gateway running 46920 tls://127.0.0.1:7557 Logs /srv/scribe/cluster-state/local/logs Binaries 0.1.0 · /srv/scribe/cluster-state/local/bin/0.1.0 CA the storage and gateway CA is being rotated since 2026-09-27; a plain renew retires the old CA on 2026-12-26 (`scribe-ctl --config /srv/scribe/scribe-cluster.toml renew --retire-ca storage` does it now) $ scribe-ctl --config ./scribe-cluster.toml renew Scribe · local · Unchanged: nothing to renew: 4 credentials are valid for 30 more days; the first to expire is the storage-a server credential on 2027-09-27 3 server certificates are valid for 30 more days; the first to expire is auth's on 2027-09-27 The storage and gateway CA is being rotated: its old CA stays trusted until 2026-12-26, when a plain renew retires it (`scribe-ctl --config /srv/scribe/scribe-cluster.toml renew --retire-ca storage` does it now). ``` From the overlap's end, the scheduled plain renew retires the old CA: each certificate is followed by the new CA itself instead of the cross certificate, `previous/` and `cross.pem` move to `retired--/` (the key is kept there, never deleted: move it offline or remove it), the trust files hold the new CA alone, the servers and the web services restart, and handshakes trusting the new CA prove them. A client that never connected during the overlap sees "CA changed" with both fingerprints and trusts the new CA by hand ([CA rotation and handover](/docs/auth.md#ca-rotation-and-handover)): ```text $ scribe-ctl --config ./scribe-cluster.toml renew --plan Scribe · local · renew plan · gateway variant · native runtime Definition /srv/scribe/scribe-cluster.toml Paths below are relative to its directory. ENDPOINTS auth tls://127.0.0.1:7443 storage-a tls://127.0.0.1:7447 gateway tls://127.0.0.1:7557 CHECKS ok credentials nothing to renew: 4 credentials are valid for 30 more days; the first to expire is the storage-a server credential on 2027-09-27 ok rotations the storage and gateway CA's overlap ended on 2026-09-27: its old CA is retired now (clients that have not connected since 2026-09-27 are asked to trust the new one) ok certificates 3 server certificates are valid for 30 more days; the first to expire is auth's on 2027-09-27 ok cluster running; storage-a, gateway restart to read the renewed credentials and certificates, one at a time while the rest serve ok permissions 10 existing secret files and auth's data root are private STEPS 1. Write storage-a's certificate followed by the new CA instead of the cross certificate into pki/storage-a/server.pem.next after the overlap, clients that trust only the old CA are no longer served 2. Write gateway's certificate followed by the new CA instead of the cross certificate into pki/gateway/server.pem.next after the overlap, clients that trust only the old CA are no longer served 3. Put storage-a's new certificate in place as pki/storage-a/server.pem, keeping the current one as pki/storage-a/server.pem.previous the server reads this file when it starts; a rollback puts the kept copy back 4. Put gateway's new certificate in place as pki/gateway/server.pem, keeping the current one as pki/gateway/server.pem.previous the server reads this file when it starts; a rollback puts the kept copy back 5. Retire the old storage and gateway CA: archive previous/ and the cross certificate as retired-2026-09-27-8ca70e3c/ in pki/ca (its key is kept, never deleted) every server presents the new CA itself from now on 6. Restart storage-a, gateway so they read their renewed files they read these files only when they start; the supervisor restarts each in turn while the others serve 7. Verify a TLS handshake with storage-a trusting the current storage and gateway CA alone clients hold one CA each; every one of them must still connect 8. Verify a TLS handshake with gateway trusting the current storage and gateway CA alone clients hold one CA each; every one of them must still connect 9. Read through every store with a reader credential, and wait for the gateway's fresh inventory every store serves reads through the new trust 10. Remove storage-a's replaced certificate pki/storage-a/server.pem.previous the new certificate is proved 11. Remove gateway's replaced certificate pki/gateway/server.pem.previous the new certificate is proved ROLLBACK (if one of steps 3-9 fails) 12. Put gateway's previous certificate back from pki/gateway/server.pem.previous into pki/gateway/server.pem the old certificate is still valid 13. Put storage-a's previous certificate back from pki/storage-a/server.pem.previous into pki/storage-a/server.pem the old certificate is still valid 14. Put the old storage and gateway CA back from retired-2026-09-27-8ca70e3c/ in pki/ca the rotation's overlap goes on 15. Restart storage-a, gateway so they read the previous files again back to the files that served before NOTES - Nothing old stops working before its replacement is proved; if a verification fails, the old files go back in place and the new credentials are revoked. Plan only: nothing was written and no lock was taken. Run renew without --plan to apply it. $ scribe-ctl --config ./scribe-cluster.toml renew local · [1/11] Write storage-a's certificate followed by the new CA instead of the cross certificate into pki/storage-a/server.pem.next… … local · [11/11] Remove gateway's replaced certificate pki/gateway/server.pem.previous… Scribe · local · Renewal complete wrote pki/storage-a/server.pem's certificate followed by pki/ca/ca.pem into pki/storage-a/server.pem.next wrote pki/gateway/server.pem's certificate followed by pki/ca/ca.pem into pki/gateway/server.pem.next put the new certificate in place as pki/storage-a/server.pem; the current one is kept as pki/storage-a/server.pem.previous put the new certificate in place as pki/gateway/server.pem; the current one is kept as pki/gateway/server.pem.previous retired the old storage and gateway CA: archived as retired-2026-09-27-8ca70e3c/ in pki/ca (keep its key offline or remove it when you no longer need it) restarted storage-a, gateway, which read their renewed files storage-a: TLS handshake at 127.0.0.1:7447 as localhost verified trusting the current storage and gateway CA alone gateway: TLS handshake at 127.0.0.1:7557 as localhost verified trusting the current storage and gateway CA alone storage-a: serves its repositories (0) from /srv/scribe/data/storage-a gateway: polled every instance storage-a: removed the replaced certificate pki/storage-a/server.pem.previous gateway: removed the replaced certificate pki/gateway/server.pem.previous RENEWED storage-a's certificate: pki/storage-a/server.pem, valid until 2027-09-27, followed by the new CA gateway's certificate: pki/gateway/server.pem, valid until 2027-09-27, followed by the new CA RETIRED The old storage and gateway CA (BLAKE3:8ca70e3c41276a59c247660fd4aa122117645038abc2e25ce8ba18560dd80f8c) is trusted nowhere any more; archived as retired-2026-09-27-8ca70e3c/ (its key is kept: move it offline or remove it). ``` `--retire-ca storage|auth` retires a rotation before its overlap ends, with that warning in its plan. With the key outside the cluster, `--ca-dir DIR` creates the new CA in `DIR/next/`, and after the run DIR follows the cluster (the new CA current, the old one in `DIR/previous/`, archived when the rotation retires with `--ca-dir DIR`); with `--remote` the new CA's public part and the certificates travel, never a key. `--rotate-ca credential|access` adds a successor to one of auth's internal client CAs, which only Scribe's services trust: the cluster drains, `scribe-auth authorities add` runs on the stopped auth root, and the cluster starts again and serves reads; the new CA signs 15 minutes after it was added. `--retire-ca credential|access` retires the oldest the same way once nothing it signed is in use (auth refuses before and says why). See [rotating the internal CAs](/docs/auth.md#rotating-the-internal-cas). ### Runs, resumes and schedulers Renew works while the cluster runs (the normal case) and under both runtimes. It takes the command lock and journals every step like setup and upgrade: an interrupted renewal resumes when run again, as a plain `renew` whatever options started it, adopting a new credential auth issued before the interruption from its `.next` (never issuing a second), a staged CA, and a rotation's switch half done; a rollback that stopped is finished from the journal alone. Other mutating commands refuse to run while a renewal is pending, and renew refuses while another one is. Crash windows inside it are tested end to end (`ctl-renewal-issued:`, `ctl-credential-kept:`, `ctl-credential-restored:`, `ctl-replaced-revoked:`, `ctl-certificate-kept:`, `ctl-key-kept:`, `ctl-key-restored:`, `ctl-next-ca-key-written:`, `ctl-ca-set-aside:`, `ctl-ca-archived:` besides the step windows; test builds only). For schedulers: renew never prompts, writes one line per step on stderr and its result on stdout, and exits 0 (renewed, or nothing due), 1 (a check or step failed, or the renewal was rolled back: the files in place work in every case) or 2 (invalid arguments). With `--json` the result is one object: `renewed` (per credential `holder`, `credential`, `file`, `replaced`, `replaced_expires_ms`, `new`, `expires_ms`), `certificates` (per file `service`, `file`, `key` (the private key a rekey replaced, else null), `change` of `reissue`, `handover`, `rechain` or `rekey`, `replaced_expires_ms`, `expires_ms`), `rotation`, `retired` and `authority`, or `unchanged: true`, or on a rollback `rolled_back`, `failure`, `rollback` and `kept`. `status --json` lists `ca_rotations`. Renew installs no schedule; the [cluster guide](/docs/cluster-usage.md#renew-service-credentials) has cron, systemd timer and Task Scheduler examples. Run it daily: it is a no-op until 30 days before an expiry or the end of an overlap. ## adopt Take over a Docker deployment scribe-ctl did not create (docker/init.sh's, or Compose projects run by hand) without moving its data or secrets: find its services, data, secrets, certificates and ports from its containers, write a definition that points at them where they are (the web interface's secrets and certificate are copied into the cluster's directory), then stop the old Compose projects (never removing them or their volumes) and start the cluster's own; a failed health check stops it and starts the old projects again as they were. Resumable ```text scribe-ctl adopt [OPTIONS] --runtime ``` | Argument or option | Meaning | | --- | --- | | `--runtime ` | Where the deployment runs: docker (Compose projects on this machine, or with --remote on that one). A native deployment is adopted with a hand-written definition and setup instead (docs/cluster-usage.md) | | `--project ` | A Compose project to take over: the one that runs Scribe's services, and again for the web interface's when it runs in another. Without it, adopt takes the one project running Scribe services that scribe-ctl does not manage, with every project on its networks, and refuses when that is ambiguous | | `--admin-credential ` | Auth's bootstrap administrator credential, which no service container mounts (docker/init.sh wrote operator/admin.credential in its state directory). Required when the deployment runs auth; it stays where it is | | `--source ` | The Scribe checkout (of this scribe-ctl's version) to build the images from, as setup does. Needed unless the running images are this scribe-ctl's release, and always with the web interface (the image set keeps its Compose file). Also builds with --plan; nothing in the deployment changes | | `--plan` | Show the plan and run every read-only check (containers, files, credentials, formats, ports, auth), without writing anything, taking a lock or touching a container | `adopt` is for a deployment that already serves: the Compose projects the removed `docker/init.sh` made (basic, auth or gateway mode), a gateway deployment with the web interface's own Compose project beside it, or the same shapes started by hand. It reads everything from `docker inspect` of the projects' containers, never from their names alone: - **Which projects.** Named with `--project` (the Scribe services' project, and again the web interface's), or found: the one Compose project that runs `scribe-auth`, `scribe-server` or `scribe-gateway` `serve` and that scribe-ctl does not run itself, plus every project with a container on one of its networks (the web interface's joins it). Two candidates are refused with both named. - **What each container is.** Auth, a store and the gateway by the process they run; the web interface's services by their Compose service being one of apps/web's (which is why the web interface needs `--source`). Any other container, a flag scribe-ctl's containers would not pass (`--session-ttl-days`, `--log-dir`, …), a storage configuration setting other than the data root, listener, auth namespace, credential and certificate, a `SCRIBE_*` variable, a second container of one service, or a data root in a named volume is refused by name: scribe-ctl never drops a setting silently. - **Where everything is.** Every container path maps to its host path through the container's bind mounts: data roots, pepper and witness, certificates and keys, each credential, the gateway's configuration and trust files, the web interface's secrets, TLS directory and reader credential. Published ports become listen addresses; a store the deployment did not publish (docker/init.sh's gateway mode) is published on loopback at its container port, so inventory and status can read it. Each certificate's CA comes from the `ca.pem` beside it (for auth, else from the credentials, which carry it); a sibling directory with that `ca.pem` and its `ca-key.pem` becomes the definition's CA directory, otherwise a copy of the certificate alone is written beside the definition and `generate_missing` is off. The definition it writes names those files where they are (absolute paths, outside the definition's directory), with `runtime = "docker"` and the variant the services form, except the web interface's secrets and certificate. Those often live in a checkout's `apps/web/deploy/local`, which may be replaced or deleted, so adopt copies them where setup keeps its own: the session key and database password to `secrets/web/`, the reader credential alone to `secrets/web-reader/scribe-web.credential`, and `fullchain.pem` and `key.pem` (when the site serves HTTPS) to `secrets/web-tls/`. The copy is a step before the old projects stop, so it adds no downtime: each file is read whole, written privately (0600, its directory 0700, owned by the definition directory's owner, as whom the containers run), synced, renamed into place and read back. The originals are never moved or changed, so the old project and a rollback keep working; a copy already there and identical is kept, and a different file where a copy goes is refused. Beside the definition scribe-ctl also writes only what it owns: `cluster-state/NAME/`, the gateway's configuration with host paths and endpoints (`scribe-gateway.toml`), the CA copies when the keys were not found, `secrets/monitor.credential` (the `scribe-monitor` inventory account, as setup provisions it) and, when the web database lived in a named volume, `data/web`. Three definition keys exist for adopted deployments: `[auth] container_port` (the port auth's credentials name as `tls://auth:PORT`, when it is not the published one), `[web] clone_host` (the host name the Clone menu gave clients) and `[web] volumes` (the web interface's named volumes kept as they are, such as `objects`). The checks, besides discovery's: the images serve the data as it is (every data root's formats equal to what the images read and write; auth schema 3 is refused because the identity v2 switch has not happened), the revision witness and pepper exist, every secret is private and so is auth's data root (auth refuses to start from a root other accounts can enter, so a mode like 0755 fails the plan with the `chmod 700 ` that fixes it, before anything is stopped), every certificate loads, every credential names auth as the cluster's containers reach it and trusts the same auth CA (an expired one is refused, one expiring within 30 days noted for `renew`), the services ran as the user scribe-ctl's containers run as (the owner of the definition's directory), ports scribe-ctl newly publishes are free, auth answers the administrator credential, and the web interface's variables would not change (its container environments compared with what `docker compose config` renders for scribe-ctl's project; `WEB_TLS` must match whether the TLS directory holds `fullchain.pem` and `key.pem`), and the web secrets to copy can be read, with no different file already where a copy goes. Then, for a gateway deployment with the web interface in its own project (`docker/init.sh gateway` with apps/web's Compose files beside it; paths shortened): ```text $ scribe-ctl --config /srv/scribe-cluster/scribe-cluster.toml --cluster prod adopt --runtime docker \ --admin-credential /srv/old-scribe/operator/admin.credential --source /srv/src --plan prod · Building scribe/auth:0.1.0 from /srv/src (the first build takes a while)… … Scribe · prod · adopt plan · gateway variant · docker runtime … CHECKS ok deployment Compose project scribe-gateway (running auth, gateway, storage-a; from /srv/scribe/docker/gateway/compose.yaml); Compose project scribe-web (running api, assets-derive, assets-derive-sandbox-image, assets-derive-sandbox-scene, assets-index, cache, content, database, nginx, objects, search, web, worker; from /srv/scribe/apps/web/compose.yaml,/srv/scribe/apps/web/compose.scribe.yaml) ok definition adopt writes cluster prod into scribe-cluster.toml ok binaries scribe-auth, scribe-server, scribe-gateway 0.1.0; setup records them as the first image set in cluster-state/prod/bin/0.1.0 ok permissions 12 existing secret files and auth's data root are private ok data in place, served as they are: auth auth_schema 4; storage-a catalog_schema 1, envelope 1, object_store 1, repository_schema 3; gateway route_store_schema 2 ok credentials 4 credentials name auth as tls://auth:7443 and trust its CA ok user the services run as 501:20, the owner of the definition's directory, as scribe-ctl's containers do ok ports the old project's published ports are free once it stops; also published now: 127.0.0.1:7447 (free) ok auth answers the administrator at tls://127.0.0.1:27443 ok web project scribe-web at http://localhost:28080; its database is copied once from the volume scribe-web_database into data/web (the volume is kept) ok web secrets 5 files to copy into secrets/web, secrets/web-reader, secrets/web-tls; the originals stay where they are … STEPS 1. Write the cluster definition scribe-cluster.toml and scribe-gateway.toml 2. Record the images scribe/auth:0.1.0, scribe/storage:0.1.0, scribe/gateway:0.1.0 and the web interface's of 0.1.0 as a set in cluster-state/prod/bin/0.1.0 3. Record 0.1.0 as the active set (bin/current) 4. Copy the web interface's secrets into secrets/web, secrets/web-reader, secrets/web-tls (5 files; the originals stay where they are) 5. Stop the old Compose projects scribe-web, then scribe-gateway (containers, networks and volumes are kept) 6. Copy the web database from the volume scribe-web_database into data/web (the volume stays as it is) 7. Start the cluster's own Compose project with 0.1.0 and wait until every service is ready 8. Ensure service account scribe-monitor reads in studio 9. Ensure scribe-monitor's credential secrets/monitor.credential 10. Read through every store with a reader credential, and wait for the gateway's fresh inventory 11. Write the cluster summary CLUSTER-INFO.txt ROLLBACK (if one of steps 5-10 fails) 12. Drain whatever of the cluster runs 13. Move data/web aside to data/web.adopt-rolled-back 14. Start the old deployment again: scribe-gateway (auth, gateway, storage-a), then scribe-web (api, assets-derive, …, worker) 15. Take cluster prod out of scribe-cluster.toml again … ``` (The step reasons are left out above.) The plan also shows the definition and the gateway file it writes as a diff. The images come from `--source` as with setup; without it, the running images are reused (named `scribe/SERVICE:BUILD` by a `docker tag` step) only when each answers `--build-info` as this scribe-ctl's release, which images older than scribe-ctl never do. The old projects are stopped with `docker compose -p NAME stop` (the web interface's first), which needs no Compose file, and are never removed: their containers, networks, images and volumes stay. The web database is copied once, stopped, by a throwaway container, compared with `diff -r` and made the cluster user's; its volume is left as it was. The health check is setup's start (`up --wait` of every container, then the gateway's fresh inventory) plus a namespace read through every store with the new inventory credential. A failure from the stop on rolls back: scribe-ctl's project stops, the database copy is moved aside (never deleted), the copies of the web secrets stay (nothing but the adopted cluster reads them; a rerun keeps them), `docker compose -p NAME start` starts exactly the services that ran before (Scribe's project first, in Compose's dependency order), and the cluster is taken out of the definition again (the file is moved aside when it defined only this cluster, else rewritten with the adopted one kept as `.bak`), so a rerun starts afresh. Verified with an injected failure after the start: ```text Scribe · prod · Adoption rolled back step 10 (Read through every store with a reader credential, and wait for the gateway's fresh inventory) failed: injected failure (ctl-step-fails:verify-serving) … ROLLED BACK drained the cluster moved data/web aside to data/web.adopt-rolled-back started scribe-gateway, scribe-web again moved the definition aside to scribe-cluster.toml.adopt-rolled-back-29d5f12ebe9ce759fb45e8f26090b6b9 The old deployment (scribe-gateway, scribe-web) runs again as it was; nothing of its data was changed. Fix the cause, then run adopt again. ``` A failure before the stop changes nothing of the deployment; the journal (`cluster-state/NAME/journal.json`) makes a rerun resume, and a rollback that stopped is finished by the next run. Once adopted, the cluster is an ordinary docker cluster: `status`, `where`, `access`, `df`, `renew` and `upgrade --source` work on it, and clients keep their endpoints, sessions and trust. The completed run lists what is left of the old deployment, for you to remove once the cluster has proven itself: ```text AFTERWARDS - Clients keep their endpoints, sessions and trust: the same ports serve the same data. … - The old Compose project scribe-gateway is stopped, not removed: its containers (scribe-gateway-gateway-1, scribe-gateway-storage-a-1, scribe-gateway-auth-1) and networks stay, and so do its files (/srv/scribe/docker/gateway/compose.yaml). Once the cluster has proven itself, remove them with `docker compose -p scribe-gateway down` (never with -v while you might still need its volumes). - … - The web database was copied out of the volume scribe-web_database, which is kept untouched; remove it with `docker volume rm scribe-web_database` once you no longer need it. - The web TLS certificate now lives in /srv/scribe-cluster/secrets/web-tls: whoever renews it must write fullchain.pem and key.pem there from now on (e.g. apps/web/deploy/issue-tls.sh --tls-dir /srv/scribe-cluster/secrets/web-tls); /srv/scribe/apps/web/deploy/local/tls is no longer read. ``` Renewing the web certificate is yours, as after setup: point whatever renews it (`apps/web/deploy/issue-tls.sh --tls-dir`, a cron job) at the new `tls_dir`. With `--remote`, adopt runs on that machine like every command (with `--source`, the checkout is pushed and the scribe-ctl that runs adopt is built there from it in Docker, named by the checkout's digest; see [remote operation](/docs/reference/cli/scribe-ctl.md#remote-operation)), and the adopted definition comes back into the local file with its `[remote]` table, as after setup; `--admin-credential` names a file on the remote machine. ## access Show who can do what in auth: every account with its kind and state, and each privilege it holds. Read-only, through the administrator credential; auth must be running ```text scribe-ctl access [OPTIONS] ``` | Argument or option | Meaning | | --- | --- | | `--user ` | Only this account (username or service account) | The report is auth's own, the one [`scribe-auth access`](/docs/reference/cli/scribe-auth.md#access) prints (scopes and access kinds are explained there), read with the cluster's `[auth] admin_credential`. It changes nothing. A cluster without auth, or with auth stopped, is refused with what to do. ```text $ scribe-ctl --config ./scribe-cluster.toml access Scribe · local · access ACCOUNT KIND STATE SCOPE ACCESS DETAIL admin person enabled auth administrator accounts, grants and credentials admin person enabled credential bootstrap user never expires scribe-gateway service enabled namespace studio reader every repository, including later ones scribe-gateway service enabled namespace studio delegate presents users' access certificates (a gateway) scribe-gateway service enabled credential scribe-ctl service-reader expires 2027-09-27 scribe-monitor service enabled namespace studio reader every repository, including later ones scribe-monitor service enabled credential scribe-ctl service-reader expires 2027-09-27 $ scribe-ctl --config ./scribe-cluster.toml access --user scribe-monitor --json { "cluster": "local", "command": "access", "ok": true, "rows": [ { "access": "reader", "account": "scribe-monitor", "detail": "every repository, including later ones", "kind": "service", "scope": "namespace studio", "state": "enabled" }, { "access": "service-reader", "account": "scribe-monitor", "detail": "expires 2027-09-27", "kind": "service", "scope": "credential scribe-ctl", "state": "enabled" } ] } ``` ## where List every file the cluster uses and what it is for, its endpoints, and the next commands ```text scribe-ctl where [OPTIONS] ``` No command-specific options; global and parent-group options still apply. `where` is read-only. It groups every file by what to do with it: trust certificates to give clients, CA private keys to back up offline, secrets that never leave the machine, the auth revision witness, configuration, data roots to back up, and runtime state. Secrets appear only as paths. It names each endpoint, where the web interface's own TLS certificate goes, the exact next commands (renewing credentials among them), and credentials or certificates within 30 days of expiry, each with what to do: `renew` for a service credential. Setup writes the same text to `CLUSTER-INFO.txt` beside the definition. Without `--cluster`, it covers every cluster in the file. Continuing the gateway example after `setup`: ```text $ scribe-ctl --config ./scribe-cluster.toml where Scribe cluster local · gateway variant Definition: /srv/scribe/scribe-cluster.toml Files below are relative to its directory. Endpoints auth tls://127.0.0.1:7443 storage-a tls://127.0.0.1:7447 gateway tls://127.0.0.1:7557 (clients connect here) Trust: public: give these to every client and to the web interface pki/ca/ca.pem storage and gateway CA certificate; clients verify servers with it (--tls-ca) pki/auth-ca/ca.pem auth CA certificate; clients verify auth with it at login (--auth-ca) CA private keys: back these up offline and never share them; setup needs them only to issue server certificates pki/ca/ca-key.pem signs storage and gateway certificates pki/auth-ca/ca-key.pem signs auth's certificate Secrets: private to this machine: never share them or put them in version control secrets/auth.pepper auth's password pepper; back it up apart from the auth data, which cannot check passwords without it secrets/admin.credential the bootstrap administrator's credential (scribe-auth --admin-credential) pki/auth/server-key.pem auth's TLS private key secrets/storage-a.credential storage storage-a's own credential secrets/monitor.credential the scribe-monitor inventory credential (df, repo list) pki/storage-a/server-key.pem storage storage-a's TLS private key secrets/gateway.credential the scribe-gateway service account's credential pki/gateway/server-key.pem the gateway's TLS private key Auth revision witness: keep it outside every auth backup; auth refuses to start behind it secrets/auth.revision auth's latest security revision Configuration: no secrets inside; keep it with your deployment scribe-cluster.toml the cluster definition scribe-gateway.toml the gateway's configuration CLUSTER-INFO.txt this summary, rewritten by setup, add and upgrade Data: back these up (docs/migration-backup-restore.md); never run two copies at once data/auth auth: accounts, credentials, grants data/storage-a storage storage-a: repositories data/gateway gateway: placement ledger Binaries: upgrade installs releases side by side and switches bin/current; it never deletes a set or a backup cluster-state/local/bin/0.1.0 active binary set: 0.1.0, selected by bin/current; services start from it Runtime: controller state and logs; keep it while the cluster exists cluster-state/local supervisor state, installed binary sets, upgrade backups, and the journal of an interrupted setup, add or upgrade cluster-state/local/logs service logs (`logs SERVICE`) Web interface The web interface's own TLS certificate is yours to provide: fullchain.pem and key.pem in its WEB_TLS_DIR (see docs/web-interface.md#connection-configuration). Its services trust /srv/scribe/pki/ca/ca.pem (WEB_SCRIBE_TLS_CA) and /srv/scribe/pki/auth-ca/ca.pem (WEB_SCRIBE_AUTH_CA). Next 1. Start the cluster (and stop it with `down`) scribe-ctl --config /srv/scribe/scribe-cluster.toml up 2. Add a person (the password comes from a private file) scribe-auth --admin-credential /srv/scribe/secrets/admin.credential users add alice scribe-auth --admin-credential /srv/scribe/secrets/admin.credential users password alice --password-file PASSWORD_FILE scribe-auth --admin-credential /srv/scribe/secrets/admin.credential namespace-admins set alice studio 3. Log in and use it scribe --server tls://127.0.0.1:7557 --tls-ca /srv/scribe/pki/ca/ca.pem login alice --auth tls://127.0.0.1:7443 --auth-ca /srv/scribe/pki/auth-ca/ca.pem scribe --server tls://127.0.0.1:7557 --tls-ca /srv/scribe/pki/ca/ca.pem repo create game 4. Check on it scribe-ctl --config /srv/scribe/scribe-cluster.toml status scribe-ctl --config /srv/scribe/scribe-cluster.toml df ``` ## example Print a complete example cluster definition to stdout ```text scribe-ctl example [OPTIONS] ``` No command-specific options; global and parent-group options still apply. ## up Start the cluster: auth, then storage, then the optional gateway (and under docker the web interface) ```text scribe-ctl up [OPTIONS] ``` | Argument or option | Meaning | | --- | --- | | `--daemon` | Detach the supervisor and return after every service is ready | | `--foreground` | Keep the supervisor attached, overriding daemon = true in the definition | `--daemon` and `--foreground` are mutually exclusive and override the definition's `daemon` (default false; `setup` writes true). Before anything starts, `up` refuses an auth data root other accounts can enter, with the `chmod 700 ` that fixes it (auth itself would refuse it after the stores started). Daemon mode returns after readiness; foreground mode stays attached until shutdown. With `--json`, foreground mode emits its final result after shutdown. Under docker, `up` rewrites the project's files and runs `docker compose up -d --wait`; containers are kept by Docker (`restart: unless-stopped`), and foreground mode stops the project on Ctrl-C. ## down Cooperatively drain all services, including children of a lost supervisor ```text scribe-ctl down [OPTIONS] ``` No command-specific options; global and parent-group options still apply. Under docker, `down` is `docker compose stop` for the project: every container gets SIGTERM and its two-minute grace period, and is kept (with its log). ## status Show supervisor and service state and expiry warnings; never probes arbitrary saved PIDs ```text scribe-ctl status [OPTIONS] ``` | Argument or option | Meaning | | --- | --- | | `--storage` | Query live disk usage and repositories on every configured storage server | Without `--cluster`, `status --storage` inspects every named cluster. Plain `status` retains its local lifecycle snapshot without network queries. It also warns about every credential and server certificate that expires within 30 days (or did), and every CA that expires within a year, read from the files themselves, with what to do about it (`scribe-ctl renew`, or for a CA `renew --rotate-ca`; see [renew](/docs/reference/cli/scribe-ctl.md#renew)), shows each CA rotation in its overlap with the day a plain renew retires it (`CA` line; `ca_rotations` with `--json`), and names the active binary set (with `--json`, `binaries`: `version`, `build` and `dir`, or null before one is recorded). When this `scribe-ctl` is another release than the set's, it says where the cluster's own is. Under docker the table lists the project's containers from `docker compose ps` (no PIDs), the web interface's too. After an upgrade: ```text $ scribe-ctl --config ./scribe-cluster.toml status Scribe · local · running SERVICE STATE PID ENDPOINT auth running 8980 tls://127.0.0.1:7443 storage-a running 8981 tls://127.0.0.1:7447 gateway running 8982 tls://127.0.0.1:7557 Logs /srv/scribe/cluster-state/local/logs Binaries 0.1.0 (build next) · /srv/scribe/cluster-state/local/bin/next This scribe-ctl is 0.1.0; the cluster's own is /srv/scribe/cluster-state/local/bin/next/scribe-ctl. ``` ## doctor Check the cluster where it runs, and this machine, without changing anything: controller, tools, ssh (remote), binaries, Docker, disk space, data formats, permissions, certificate and credential expiry, services and ports, interrupted commands, backups and adoption leftovers. Each check is ok, info, warn or fail with a fix. Exit 0 when nothing warns or fails, 1 with warnings only, 3 with any failure (2 stays a usage error). Writes, locks, starts, pushes and installs nothing ```text scribe-ctl doctor [OPTIONS] ``` No command-specific options; global and parent-group options still apply. `doctor` is the health check to run first when something seems wrong, and from a monitor on a schedule. It observes the cluster as every plan does (the same discovery, the runtime's status table, the files' own contents) and judges each thing it sees by the rule the mutating commands plan with, so its verdicts and theirs cannot drift: the owner-only rule auth applies to its data root, the upgrade's free-space reserve, the format comparison `upgrade` makes, the 30-day expiry window `status` and `renew` use. It never writes a file (not even a lock file), never starts, stops or restarts a service, and never repairs anything; each problem comes with the command or change that fixes it. Repository integrity is `scribe-server doctor`'s job, offline ([routine health checks](/docs/operations.md#routine-health-checks)). Checks are grouped in sections, in this order; each has a stable id that a monitor can key on (`<…>` is a service, store or project name): | Section | Checks | | --- | --- | | Controller | `ctl` (this build), `definition` (it parses and selects the cluster), `definition.files` (every file it names exists; for a cluster that runs here), and for a remote cluster `tool.ssh`, `tool.tar`, `tool.rsync` (rsync 3, or openrsync and 2.x, which push as tar over ssh) | | SSH and the remote | Remote clusters only: `ssh.reach` (with the time to connect and log in), `ssh.login` (key login in batch mode, never a prompt), `ssh.shared` (later commands share the connection), `remote.host`, `remote.directory`, `remote.definition` (the remote copy against this machine's, which the next command pushes), `remote.ctl` (the remote scribe-ctl against this one), `remote.doctor` | | Binaries | `binaries.active` (`bin/current` and its set), `binaries.controller` (this scribe-ctl against the cluster's release), `binaries.installed` | | Docker | Docker clusters only: `docker.engine`, `docker.compose`, `docker.images` (every image of the active set), `docker.build-cache` (info, a warning above 20 GiB) | | Disk space | `disk.backups` (room for an upgrade's backups of every root, by the upgrade's own rule: the copy and max(10 %, 1 GiB) beside it) and `disk.` for auth, each store and the web database (a warning below that reserve) | | Data | `data.`: exists, initialized, bound to its namespace, and records the formats the active set reads and writes (newer data or a pending migration fails) | | Secrets and permissions | `permissions.auth-root` (auth's own owner-only rule; the fix is the `chmod 700`), `permissions.secrets`, `auth.witness` and `auth.pepper` (present, and outside auth's data root by auth's placement rule) | | Certificates and credentials | `expiry.` for every credential, certificate and CA (a warning within 30 days, for a CA within a year, a failure once expired, with what replaces it); `ca.rotation.` for a CA rotation (info in its overlap with the day it retires, a warning once overdue, for a staged `next/` no run uses, or a leftover `previous/`); `ca.key.` for each server CA's key (a warning while it is a classical key such as ECDSA P-256 rather than ML-DSA, with the `renew --rotate-ca` that replaces it); `ca.internal.` for auth's internal CAs from its trust bundle while it runs (a warning within a year of expiry), and `ca.internal..key` when one of them still has a classical key (rotate the one that signs, retire an older one still trusted); `ca.auth-shared` (info) when `[auth] ca` is the storage and gateway CA, whose expiry is then auth's; `renew.leftovers.` (a warning for a `.next` or `.previous` file beside a server's certificate or key with no renewal pending; a staged `.next` is what `renew --rekey ` uses); and with the web interface `web.tls`: its certificate is there for an `https://` origin and names the origin's host | | Services | `cluster`, `service.` (under docker every container of the project with its health check, the web interface's included), `port.` (accepts a connection while running; free while stopped), `gateway.inventory` (fresh from every instance) | | Operations | `journal` (an interrupted command, with the command that finishes it), `backups` (info), and under docker `adopt.leftovers`, `adopt.project.` (an old Compose project that still mounts the cluster's data: warned while stopped, failed while it runs) and `adopt.volume.` (the old web database volume adopt copied) | `cluster` counts every service the cluster runs as the runtime lists it: under docker with the web interface, every container of the project, Scribe's and the web interface's apart (`running (16 services: 3 Scribe, 13 web)`). A stopped cluster warns (`cluster` and each `service.`, fixed by `up`); a port another process holds, a service that exited or reports unhealthy, or a gateway without fresh inventory fails. A definition that does not read, or a cluster that cannot be selected, is a failed `definition` check, not an error: doctor always reports. With a remote cluster (`--remote` or `[remote]`), this machine checks its tools and ssh, probes the remote as every remote command does, and runs the remote's own scribe-ctl `doctor --json` in the cluster's directory; its sections join the report after this machine's, marked `on HOST:DIR`, with one summary and one exit code. Nothing is pushed or installed: a remote without a scribe-ctl that answers is a failed `remote.ctl` check naming the command that installs one, and one of another release is reported (`remote.ctl` warns) and still asked. The only thing doctor creates is ssh's control socket, in a private directory under `/tmp` removed when it ends. A cluster on another machine (its definition has a `[remote]` table): ```text $ scribe-ctl --config ./scribe-cluster.toml doctor Scribe · local · doctor · native runtime · on deploy@scribe-host:scribe Definition /home/me/scribe/scribe-cluster.toml Read only: nothing is written, locked, started or stopped. … SSH AND THE REMOTE ok ssh.reach deploy@scribe-host:scribe answered and logged in within 0.03 s ok ssh.login key login in batch mode: no password or host-key prompt ok ssh.shared later commands share the one connection (ControlMaster) ok remote.host macos/aarch64; rsync 3 no; docker no ok remote.directory scribe exists ok remote.definition scribe/scribe-cluster.toml holds cluster local as this machine's definition does ok remote.ctl bin/0.1.0/scribe-ctl is this release, 0.1.0 ok remote.doctor the remote's scribe-ctl 0.1.0 checked the cluster there … SERVICES · on deploy@scribe-host:scribe ok cluster running (1 service) ok service.storage-a running, pid 75477, tcp://127.0.0.1:47547 ok port.storage-a 127.0.0.1:47547 accepts connections (0.2 ms) … Healthy · 27 ok · 1 info · 0 warn · 0 fail · exit 0 ``` `remote.ctl` is ok when the remote runs the scribe-ctl every remote command from here would run (the rule of [Remote operation](/docs/reference/cli/scribe-ctl.md#remote-operation)): this release exactly, or, from a machine of another OS or architecture, the one `bin/controller` or `bin/current` names. Its fix names what actually works from here. On the remote's own platform, any command run with `--remote` installs this machine's build (as does `bootstrap --binaries DIR` with a release directory of it); from another platform only [bootstrap](/docs/reference/cli/scribe-ctl.md#bootstrap) can, and it does not touch the cluster: ```text SSH AND THE REMOTE … FAIL remote.ctl no scribe-ctl of this release answers in scribe/bin fix: this machine's scribe-ctl cannot run on the remote (linux/x86_64, this is macos/aarch64): `scribe-ctl --config /home/me/scribe/scribe-cluster.toml bootstrap --source ` builds this release's there in Docker from a checkout of 0.1.0, without touching the cluster; doctor pushes nothing ``` A remote without docker gets `bootstrap --binaries DIR` (this release built for the remote) instead. The same fix follows a `remote.doctor` failure, a remote scribe-ctl too old to know `doctor`. Exit status: 0 when every check is ok or info, 1 when the worst is a warning, 3 when any check fails. 2 stays clap's usage error, as for every command, so a scheduler can tell a sick cluster from a mistyped command. `--json` prints one document (see [JSON output](/docs/json-output.md#controller-results)); expiry checks add `expires` with `at` and `days_left`, so a monitor needs no text parsing: ```sh scribe-ctl --config ./scribe-cluster.toml --json doctor \ | jq -r '.sections[].checks[] | select(.expires) | "\(.expires.days_left)\t\(.id)"' | sort -n ``` A healthy cluster: ```text $ scribe-ctl --config ./scribe-cluster.toml doctor Scribe · local · doctor · native runtime Definition /srv/scribe/scribe-cluster.toml Read only: nothing is written, locked, started or stopped. CONTROLLER ok ctl scribe-ctl 0.1.0 for macos/aarch64 ok definition /srv/scribe/scribe-cluster.toml defines cluster local (gateway variant) under runtime native ok definition.files all 18 files and directories it names exist BINARIES ok binaries.active 0.1.0 for macos/aarch64 in cluster-state/local/bin/0.1.0 ok binaries.controller this scribe-ctl is the cluster's release, 0.1.0 ok binaries.installed 1 set installed DISK SPACE ok disk.backups room for an upgrade's backups: 1.18 MiB to copy to cluster-state/local/backups, 422.20 GiB available ok disk.auth 422.20 GiB available where data/auth holds 1.01 MiB ok disk.storage-a 422.20 GiB available where data/storage-a holds 177.33 KiB DATA ok data.auth data/auth: auth_schema 4, as the active scribe-auth reads and writes ok data.storage-a data/storage-a: catalog_schema 1, envelope 1, object_store 1, repository_schema 3, as the active scribe-server reads and writes ok data.gateway data/gateway: route_store_schema 2, as the active scribe-gateway reads and writes SECRETS AND PERMISSIONS ok permissions.auth-root data/auth is reachable by its owner only ok permissions.secrets 10 secret files are private to their owner ok auth.witness secrets/auth.revision, outside auth's data root ok auth.pepper secrets/auth.pepper, outside auth's data root CERTIFICATES AND CREDENTIALS ok expiry.administrator-credential administrator credential (secrets/admin.credential): does not expire ok expiry.auth-certificate auth certificate (pki/auth/server.pem): expires in 364 days (2027-09-27) ok expiry.storage-a-server-credential storage-a server credential (secrets/storage-a.credential): expires in 364 days (2027-09-27) ok expiry.inventory-credential inventory credential (secrets/monitor.credential): expires in 364 days (2027-09-27) ok expiry.storage-a-certificate storage-a certificate (pki/storage-a/server.pem): expires in 364 days (2027-09-27) ok expiry.gateway-credential gateway credential (secrets/gateway.credential): expires in 364 days (2027-09-27) ok expiry.gateway-certificate gateway certificate (pki/gateway/server.pem): expires in 364 days (2027-09-27) ok expiry.storage-and-gateway-ca storage and gateway CA (pki/ca/ca.pem): expires in 3649 days (2036-09-24) ok expiry.auth-ca auth CA (pki/auth-ca/ca.pem): expires in 3649 days (2036-09-24) SERVICES ok cluster running (3 services) ok service.auth running, pid 70911, tls://127.0.0.1:47443 ok service.storage-a running, pid 70943, tls://127.0.0.1:47447 ok service.gateway running, pid 70960, tls://127.0.0.1:47557 ok port.auth 127.0.0.1:47443 accepts connections (0.2 ms) ok port.storage-a 127.0.0.1:47447 accepts connections (0.1 ms) ok port.gateway 127.0.0.1:47557 accepts connections (0.1 ms) ok gateway.inventory the gateway has fresh inventory from every instance OPERATIONS ok journal no interrupted command ok backups no upgrade backups kept Healthy · 35 ok · 0 info · 0 warn · 0 fail · exit 0 ``` The same cluster stopped, with auth's data root opened to other accounts: ```text $ scribe-ctl --config ./scribe-cluster.toml doctor … SECRETS AND PERMISSIONS FAIL permissions.auth-root /srv/scribe/data/auth is accessible to other accounts (mode 755) and holds private keys; restrict it with chmod 700 /srv/scribe/data/auth fix: chmod 700 /srv/scribe/data/auth ok permissions.secrets 10 secret files are private to their owner ok auth.witness secrets/auth.revision, outside auth's data root ok auth.pepper secrets/auth.pepper, outside auth's data root SERVICES WARN cluster stopped fix: `scribe-ctl --config /srv/scribe/scribe-cluster.toml up` starts it WARN service.auth stopped fix: `scribe-ctl --config /srv/scribe/scribe-cluster.toml up` starts the cluster WARN service.storage-a stopped fix: `scribe-ctl --config /srv/scribe/scribe-cluster.toml up` starts the cluster WARN service.gateway stopped fix: `scribe-ctl --config /srv/scribe/scribe-cluster.toml up` starts the cluster ok port.auth 127.0.0.1:47443 is free for the next start ok port.storage-a 127.0.0.1:47447 is free for the next start ok port.gateway 127.0.0.1:47557 is free for the next start … Failed checks: see the fixes above · 29 ok · 0 info · 4 warn · 1 fail · exit 3 ``` ## df Show live disk usage for every storage server (apparent bytes) ```text scribe-ctl df [OPTIONS] ``` Alias: `usage`. No command-specific options. Select `--cluster NAME` when multiple clusters are defined. ## repo list List every repository with its storage server, head and size ```text scribe-ctl repo list [OPTIONS] ``` No command-specific options. Select `--cluster NAME` when multiple clusters are defined. Repository names are grouped by storage server; equal names on different servers remain separate entries. ### Live inventory behavior All three commands query the existing paginated storage inventory API. They include configured storage services and additional gateway backends. An exact endpoint match between a storage service and gateway backend is queried once, under the storage service name. The gateway backend's CA is reused for that match unless `storage.ca` overrides it. The gateway need not be running. Port-zero local stores use their recorded endpoint only while their data-root lock is held. Authenticated storage needs `storage.monitor_credential`, the credential file of a service account with a namespace reader grant in the store's namespace, and trusts `storage.ca`, the matching gateway backend's CA, or `certificates.ca_dir/ca.pem`. The store's server credential is not a user credential and is never used for inventory. An authenticated gateway backend with no matching storage service has no monitor credential and is reported as a per-server error. Inventory never provisions credentials or changes permissions. See the [cluster guide](/docs/cluster-usage.md#live-storage-inventory) for configuration. Terminal inventory uses aligned tables and readable IEC sizes (KiB, MiB, GiB, and larger units). Use `--json` for automation: size fields remain exact integer byte counts, with no rounding or unit suffixes. Used bytes are apparent data-root sizes, sampled live rather than an atomic cluster snapshot. Available bytes are shown per server and never added together: several roots can share a filesystem. Totals cover responding configured targets; alias endpoints pointing at the same store should not be configured twice. This is the online inventory, not the detailed offline `scribe-server usage` object breakdown. Queries use up to four connections at once per cluster and a 10-second connect/I/O timeout (the gateway's `backend_timeout_secs` when configured). Every page is checked for ordered repository IDs, stable instance identity and repository count. Retry if repositories change during collection. Unreachable, unauthorized or invalid responses appear as per-server errors; successful results remain visible, totals are marked partial, and exit status is 1. JSON emits one result with `ok`, `command`, `cluster`, and `clusters`. Each cluster contains `servers`, `totals`, and `lifecycle` (only populated for `status --storage`). Failed servers have null size fields and an `error`. ## logs Show the last lines of a service or supervisor log ```text scribe-ctl logs [OPTIONS] ``` | Argument or option | Meaning | | --- | --- | | `` | Service name (auth, gateway, or a storage name), or supervisor; under docker any service of the Compose project (the web interface's too) | | `--lines ` | Number of lines, reading at most the final 64 KiB [default: 40] | `--lines` defaults to 40, accepts 1–1000, and reads at most the final 64 KiB. SERVICE is `auth`, `gateway`, a configured storage name, or `supervisor`. This prints a bounded tail; it does not follow the log. Under docker it is `docker compose logs --no-log-prefix --tail LINES SERVICE`, for any service of the project (`api`, `worker`, … of the web interface too). ## ssh Open a shell in the remote cluster's directory, or run a command there (after --). Needs --remote or a [remote] table ```text scribe-ctl ssh [OPTIONS] [-- ...] ``` | Argument or option | Meaning | | --- | --- | | `[COMMAND]...` | The command and its arguments, each passed as one word | Without a command it opens a login shell (with a terminal) in the cluster's directory on the remote; there, `bin/controller` names the newest scribe-ctl installed (`bin//scribe-ctl --config status` runs the cluster's commands directly, on the copy of the definition). With one, each word reaches the remote command exactly as given (quoted for the remote shell), in that directory, and its exit code is this command's. Nothing is pushed or installed. ```text $ scribe-ctl --config ./scribe-cluster.toml ssh -- ls CLUSTER-INFO.txt bin cluster-state data pki scribe-cluster.toml secrets $ scribe-ctl --config ./scribe-cluster.toml ssh -- sh -c 'exit 7'; echo "exit $?" exit 7 ``` ## bootstrap Install or refresh the scribe-ctl that runs this cluster's commands on its remote machine, without touching the cluster: built there in Docker from a checkout (--source; for a remote of another OS or architecture, which this machine's scribe-ctl cannot run on) or from a release directory built for the remote (--binaries). It is named in bin/controller, which later commands from a machine whose own build cannot run there use; bin/current, the definition there, services, images, lock and journal stay as they are, so it is safe beside a command running there. Needs --remote or a [remote] table ```text scribe-ctl bootstrap [OPTIONS] ``` | Argument or option | Meaning | | --- | --- | | `--source ` | A Scribe checkout of this scribe-ctl's version: what its scribe-ctl build reads is pushed to `upload/controller/source` there and built in Docker (`docker/Dockerfile.ctl`) as build `-src.`. Needs docker on the remote. Nothing is pushed or built when that build is there already | | `--binaries ` | A release directory of this scribe-ctl's version built for the remote's OS and architecture: its executables are pushed to `upload/controller/binaries` and installed as `bin//` there | | `--plan` | Show what would be pushed, built and installed, after probing the remote, without changing anything there | A scribe-ctl here can only install its own build on a remote of the same OS and architecture, which every remote command does by itself. From a laptop of another platform (macOS, say, and a Linux server), the remote runs the scribe-ctl the last `--source` command built there, which knows only the commands of its checkout. `bootstrap --source` brings a newer one over without an upgrade: it pushes the checkout (the same selection as any `--source` push, into `upload/controller/source`, so the `upload/source` of a command running there is left alone), builds only scribe-ctl there, installs it as `bin//` and names it in `bin/controller`. Nothing else changes: not `bin/current`, the definition there, the cluster's binary set or images, its services, lock or journal, so it takes no lock and may run while an upgrade runs there (an existing `bin/` that answers as that build is never replaced). Rules 3 and 4 of [Remote operation](/docs/reference/cli/scribe-ctl.md#remote-operation) then pick it for every command from a machine whose own build cannot run there, `doctor` included; a machine of the remote's platform keeps installing and running its own build. A later `--source` setup, upgrade or adopt still names its own build in `bin/current` and `bin/controller`. ```text $ scribe-ctl --config ./scribe-cluster.toml --remote deploy@scribe-host:scribe bootstrap --source ../scribe --plan Checking deploy@scribe-host:scribe over ssh… Scribe · deploy@scribe-host:scribe · Controller plan Would push the checkout ../scribe to scribe/upload/controller/source (1812 entries, 13.0 MB) Would build scribe-ctl 0.1.0 (build 0.1.0-src.25b366157b78) for linux/x86_64 there in Docker (docker/Dockerfile.ctl) Would install it as bin/0.1.0-src.25b366157b78/scribe-ctl, named in bin/controller Would leave the cluster as it is: bin/current (0.1.0 (build 0.1.0-src.3fa9c2e1b7d0)), its services, images, lock and journal Plan only: nothing was pushed, built or installed. $ scribe-ctl --config ./scribe-cluster.toml --remote deploy@scribe-host:scribe bootstrap --source ../scribe Checking deploy@scribe-host:scribe over ssh… Pushing the checkout ../scribe to scribe/upload/controller/source (1812 entries, 13.0 MB); tar over ssh… Building scribe-ctl 0.1.0 (build 0.1.0-src.25b366157b78) for linux/x86_64 there in Docker (docker/Dockerfile.ctl); the first build takes a while… Scribe · deploy@scribe-host:scribe · Controller installed Pushed the checkout ../scribe to scribe/upload/controller/source (1812 entries, 13.0 MB) Built scribe-ctl 0.1.0 (build 0.1.0-src.25b366157b78) for linux/x86_64 there in Docker (docker/Dockerfile.ctl) Installed it as bin/0.1.0-src.25b366157b78/scribe-ctl, named in bin/controller Left the cluster as it is: bin/current (0.1.0 (build 0.1.0-src.3fa9c2e1b7d0)), its services, images, lock and journal Remote commands from a machine whose own scribe-ctl cannot run on linux/x86_64 now run bin/0.1.0-src.25b366157b78/scribe-ctl. ``` Run again with the same checkout, it pushes and builds nothing (`Controller unchanged`, `bin//scribe-ctl is there already`). `--binaries DIR` pushes a release directory built for the remote instead (checked there: its scribe-ctl must answer `--build-info` for the remote's OS and architecture and this version) and installs its executables. Both refuse another version than this scribe-ctl's, since remote runs from here only pick this version. Locally, without `--remote` or a `[remote]` table, it is refused. `--json` returns `remote`, `plan`, `build`, `target`, `controller`, `unchanged`, `current` and `steps`. ## Internal supervisor entry point `up --daemon` launches the hidden `supervise` action. It is an implementation entry point; use `up` for normal operation so readiness and lifecycle state are coordinated by the controller. ```text scribe-ctl supervise [OPTIONS] --generation ``` | Argument or option | Meaning | | --- | --- | | `--generation ` | Supervisor run identifier generated by `up`, matching readiness to this run | The global configuration and cluster selectors apply here too. This action is intentionally omitted from the public help command list. ## Use cases ### Set up a new cluster ```sh scribe-ctl --config ./scribe-cluster.toml setup --variant gateway --plan scribe-ctl --config ./scribe-cluster.toml setup --variant gateway scribe-ctl --config ./scribe-cluster.toml where scribe-ctl --config ./scribe-cluster.toml up ``` ### Grow a storage cluster ```sh scribe-ctl --config ./scribe-cluster.toml down scribe-ctl --config ./scribe-cluster.toml add auth --plan scribe-ctl --config ./scribe-cluster.toml add auth scribe-ctl --config ./scribe-cluster.toml add gateway scribe-ctl --config ./scribe-cluster.toml up ``` ### Upgrade to a new release ```sh scribe-ctl --config ./scribe-cluster.toml upgrade --binaries /opt/scribe/0.2.0 --plan scribe-ctl --config ./scribe-cluster.toml upgrade --binaries /opt/scribe/0.2.0 scribe-ctl --config ./scribe-cluster.toml status ``` ### Run a cluster in Docker, with the web interface ```sh # from the checkout: build and install scribe-ctl, then set up from the same checkout ./scripts/install.sh --setup gateway --config ~/scribe/scribe-cluster.toml -- --runtime docker --web --source . --plan ./scripts/install.sh --setup gateway --config ~/scribe/scribe-cluster.toml -- --runtime docker --web --source . scribe-ctl --config ~/scribe/scribe-cluster.toml up # later, after pulling new source (the build is named by the commit, or SCRIBE_BUILD_ID): ./scripts/install.sh --upgrade --config ~/scribe/scribe-cluster.toml -- --source . --plan ./scripts/install.sh --upgrade --config ~/scribe/scribe-cluster.toml -- --source . ``` ### Renew credentials before they expire ```sh scribe-ctl --config ./scribe-cluster.toml renew --plan scribe-ctl --config ./scribe-cluster.toml renew # from a scheduler, daily; a no-op until 30 days before an expiry scribe-ctl --config /srv/scribe/scribe-cluster.toml renew --json ``` ### Take over an existing Docker deployment ```sh scribe-ctl --config ./scribe-cluster.toml adopt --runtime docker --admin-credential /srv/old-scribe/operator/admin.credential --source . --plan scribe-ctl --config ./scribe-cluster.toml adopt --runtime docker --admin-credential /srv/old-scribe/operator/admin.credential --source . scribe-ctl --config ./scribe-cluster.toml status ``` ### Operate a cluster on another machine ```sh scribe-ctl --config ./prod.toml --cluster prod --remote deploy@scribe-host:/srv/scribe setup --variant gateway --binaries ./linux-x86_64 --plan scribe-ctl --config ./prod.toml --cluster prod --remote deploy@scribe-host:/srv/scribe setup --variant gateway --binaries ./linux-x86_64 scribe-ctl --config ./prod.toml up scribe-ctl --config ./prod.toml status scribe-ctl --config ./prod.toml upgrade --binaries ./linux-x86_64-next scribe-ctl --config ./prod.toml renew scribe-ctl --config ./prod.toml ssh # from a laptop of another platform: refresh the scribe-ctl there, nothing else scribe-ctl --config ./prod.toml bootstrap --source . ``` ### Inspect a managed cluster ```sh scribe-ctl --config ./scribe-cluster.toml doctor scribe-ctl --config ./scribe-cluster.toml status --storage --json scribe-ctl --config ./scribe-cluster.toml df scribe-ctl --config ./scribe-cluster.toml repo list scribe-ctl --config ./scribe-cluster.toml access scribe-ctl --config ./scribe-cluster.toml logs supervisor --lines 50 ``` Add `--cluster NAME` when selection is required. For a guided walk through setup, `add`, and everyday operation, follow [the tutorials](/docs/tutorials.md) ([4. Operate the cluster](/docs/tutorials/04-use-ctl.md) covers these commands). The [cluster guide](/docs/cluster-usage.md) covers configuration and lifecycle rules. ## Output and exit codes `doctor` exits 0 when every check passes, 1 with warnings only and 3 with a failed check. Otherwise, exit 0 means success, 1 means runtime failure (including a `setup --plan`, `add … --plan`, `upgrade --plan`, `renew --plan` or `adopt --plan` whose checks fail, and an upgrade, renewal or adoption that was rolled back), and 2 means invalid CLI arguments. `renew` with nothing due exits 0. An upgrade handed over to a newer `scribe-ctl` exits with that controller's exit code, a remote command with the remote scribe-ctl's (255 when ssh itself failed), and `ssh -- COMMAND` with the command's. Long-running foreground commands report their final JSON result after shutdown. `--json` selects machine-readable stdout results; diagnostics stay separate. See [JSON output](/docs/json-output.md) for schemas, help, and error behavior, and [logging](/docs/logging.md) for filters, sinks, timing events, and build features.