scribe-ctl CLI reference

Scribe's operations tool.

scribe-ctl sets up and operates named clusters from one definition file. It runs an auth service, storage services, and a gateway as supervised local processes (the native runtime), or as containers of one Docker Compose project it generates (the docker runtime, optionally with the web interface), and creates what they need: data roots, both server-certificate CAs, certificates, the auth pepper and revision witness, and every service credential. It starts services in dependency order and drains them in reverse order.

Every mutating command discovers what exists, plans typed steps, and executes exactly that plan; --plan shows the plan and stops. setup provisions missing resources and is safe to rerun; add auth and add gateway grow a cluster by one service; upgrade moves it to a new release with backups, explicit migrations and rollback; renew replaces credentials and server certificates before they expire, proving each new one before the old goes, and rotates a CA through a signed handover; up requires a set-up cluster. With --remote (or a [remote] table) every command runs on the machine that holds the cluster, over ssh; bootstrap refreshes the scribe-ctl there without touching the cluster. Daemon mode detaches the supervisor but does not install a boot-time service or automatic restart policy. See the cluster guide for the definition, variants, and lifecycle workflows.

scribe | scribe-server | scribe-auth | scribe-gateway | scribe-runner

Reading this reference

Command synopses below come from this checkout’s built --help output. Angle brackets denote values, square brackets optional arguments, and ... repeatable arguments. [OPTIONS] includes the global options below and any command-specific options. Global options work before or after a subcommand; parent-group options also apply to their child commands. Use -- to end option parsing when passing a positional path that starts with a dash.

Run scribe-ctl --help for the command list, scribe-ctl COMMAND --help for command help, and scribe-ctl --version for the build version. scribe-ctl help COMMAND also displays command help. The command sections below cover every public action.

Help and version

-h / --help works at every command level. -V / --version is a top-level option; place it directly after the executable name.

Global options

Argument or option	Meaning
`--config <FILE>`	Cluster definition; defaults to scribe-cluster.toml beside this executable
`--cluster <NAME>`	Named cluster; required when the file defines several (status and where show all)
`--timings`	Report controller phase timings to stderr
`--log-level <LOG_LEVEL>`	Diagnostic verbosity: off, error, warn, info, debug, or trace
`--log-format <LOG_FORMAT>`	Diagnostic output format: text or json
`--json`	Emit one JSON result on stdout; diagnostics remain on stderr
`--remote <TARGET>`	Run the command on the machine that holds the cluster, over the system ssh (keys, agent and ssh config only; never a password prompt): `[USER@]HOST:DIR`, `ssh://[USER@]HOST[:PORT]/DIR`, or `HOST` alone with the directory from the definition's `[remote]`. Overrides the cluster's `[remote]` table

Defaults and configuration

Without --config, the controller reads scribe-cluster.toml beside its own executable. Relative paths in the definition resolve beside that file. With multiple clusters, select --cluster NAME for setup, lifecycle commands, and logs; status and where without a selector report all clusters. setup of a new definition names its cluster local unless --cluster says otherwise. example does not load configuration; it prints the shipped template.

Services, their one-shot tools and the detached supervisor run from the cluster's active binary set, <state_dir>/bin/<build>/, which <state_dir>/bin/current selects (a file holding the build id, not a symlink). Until setup records the first set, they come from the definition's [binaries] paths, or from beside this scribe-ctl. Under docker the set is images instead, built from a checkout (see Docker runtime). scribe-ctl --build-info answers with its own build and state format, like the service executables.

Logging defaults to warn, on stderr. --log-level accepts off, error, warn, info, debug, or trace; SCRIBE_LOG also supplies a target filter. --log-format accepts text or json. There are no --log-filter, verbose, or quiet switches. --timings requires the default-enabled timings feature. --json selects command results independently of diagnostics.

A cluster whose definition has a [remote] table (or any command given --remote) runs on that machine: see Remote operation. example always runs here.

Command index

setup — Create or complete a cluster: definition, data roots, CAs, certificates, secrets and credentials
add auth — Put auth in front of a storage-only cluster; repositories are kept
add gateway — Put a TLS gateway in front of an auth cluster's stores
upgrade — Upgrade the cluster to a new release (executables, or images built from a checkout), with backups, explicit migrations, a health check and rollback
prune — Remove what old releases left behind, and only what scribe-ctl made: older binary sets, their unused images, stray remote scribe-ctl builds, older upgrade backups
renew — Replace every credential and server certificate that expires within 30 days, proving each before the old one goes; retire a CA rotation's old CA after its overlap; --rotate-ca starts a rotation; --rekey replaces servers' private keys
adopt — Take over a Docker deployment scribe-ctl did not create, its data and secrets left where they are, with a rollback to the old Compose projects
access — Show who can do what in auth: every account, its kind and state, and its privileges
where — List every file the cluster uses and what it is for, its endpoints, and the next commands
example — Print a complete example cluster definition to stdout
up — Start the cluster: auth, then storage, then the optional gateway (and under docker the web interface)
down — Cooperatively drain all services, including children of a lost supervisor
status — Show supervisor and service state and expiry warnings
doctor — Check the cluster where it runs, and this machine, without changing anything; exit 0, 1 (warnings) or 3 (failures)
df — Show live disk usage for every storage server
repo list — List every repository with its storage server, head and size
logs — Show the last lines of a service or supervisor log
ssh — Open a shell in the remote cluster's directory, or run a command there
bootstrap — Install or refresh the scribe-ctl that runs a remote cluster's commands, without touching the cluster

setup

Create or complete a cluster: definition, data roots, CAs, certificates, secrets and credentials. Resumable; safe to rerun

text

scribe-ctl setup [OPTIONS]

Argument or option	Meaning
`--variant <VARIANT>`	What a new definition describes: `storage` (one plaintext server), `auth` (auth and one TLS server) or `gateway` (auth, one TLS server and a gateway). With an existing definition, it must match
`--namespace <NAME>`	Auth namespace of the storage server in a new auth or gateway definition [default: studio]
`--port-offset <N>`	Add this to every default port of a new definition (auth 7443, storage 7447, gateway 7557, web 8080 and 8082) [default: 0, or the next free multiple of 100 beside other clusters in the file]
`--runtime <RUNTIME>`	Where a new definition's services run: native (local processes) or docker (a Compose project scribe-ctl generates) [default: native]
`--web`	With a new docker definition: also run the web interface (apps/web) in the cluster's Compose project
`--source <DIR>`	Under docker: the Scribe checkout (of this scribe-ctl's version) to build the images from, tagged with this scribe-ctl's build. Also builds with --plan; nothing in the cluster changes
`--binaries <DIR>`	With --remote: this release's executables built for the remote machine (scribe-ctl, scribe-auth, scribe-server, scribe-gateway), pushed and installed there beside its scribe-ctl, which setup records. Not needed when this machine's OS and architecture match the remote's (this scribe-ctl's own directory is pushed) or under docker with --source (scribe-ctl is built there)
`--plan`	Show the plan and run every read-only check, without writing anything or taking a lock

When the file does not define the cluster, --variant is required and setup writes the definition first (with an explanatory comment on every setting), plus scribe-gateway.toml for the gateway variant. Paths are relative to the definition; services listen on loopback. A file that already defines other clusters is rewritten canonically, the previous one kept as <file>.bak; the new cluster's paths start with its name and its ports move clear of the others. When the file defines the cluster, setup uses it as written and refuses a --variant it does not match; --namespace and --port-offset are refused, and so are a --runtime other than the definition's (setup never moves a cluster between runtimes) and --web for a definition without [web].

Setup then creates whatever is missing, in dependency order: both CAs (certificates.ca_dir for storage and gateway, certificates.auth_ca_dir for auth) and each server certificate when certificates.generate_missing is on; the gateway ledger; auth with its pepper, revision witness and bootstrap administrator credential (scribe-auth init); each store, bound to its auth namespace. With auth running (started temporarily when the cluster is stopped) it then ensures:

each authenticated store is registered and holds a live server credential;
for a TLS gateway, the scribe-gateway service account with namespace reader and delegate grants in every namespace it fronts, and its credential at the gateway's [tls] credential path;
for every monitor_credential, the scribe-monitor service account with a namespace reader grant and that credential;
every user in accounts_file.

A credential is issued when its file is missing and kept otherwise, also near or past its expiry: setup never replaces one. Its plan names each service credential within 30 days of expiry and points to renew, which replaces it. A file of another holder, or a revoked one, is refused. Last, setup writes CLUSTER-INFO.txt beside the definition (see where).

Setup also records the executables the cluster runs, so that an upgrade has a set to return to. When the state directory has no bin/current yet, it takes them from the definition's [binaries] paths, or from beside this scribe-ctl (scribe-ctl itself included), checks that they are one release for this machine (--build-info), copies them side by side into <state_dir>/bin/<build>/ and writes bin/current. From then on every service, tool and the detached supervisor starts from that set; [binaries] is not read again. A cluster set up before sets were recorded gets one when setup runs again, even while it runs.

Before changing anything setup runs its checks: the definition, binaries (the recorded set must have every executable the cluster runs), private permissions on existing secrets and an existing auth data root (auth refuses to start from a root other accounts can enter; the check names the chmod 700 that fixes it), the accounts file, whether the cluster runs, free listen ports, certificate and data-root consistency. A failed check refuses the whole run. Existing state setup did not make, such as a partial PEM pair or a data root without its format marker, is refused rather than replaced. While the cluster runs, setup performs only the identity steps through the running auth (for example new accounts).

Setup records each step in a journal in the state directory before and after running it. An interrupted setup resumes when run again: a step it had started is redone, and its partial output is first moved aside beside the original as <name>.interrupted-<run>, never deleted. If an interrupted run left auth running, status shows it as unmanaged; run down before setup again.

--plan prints the same plan (checks, steps with the reason for each, notes) and exits 1 when a check fails. It writes nothing, takes no lock and starts nothing. For example, for a new gateway cluster:

text

$ scribe-ctl --config ./scribe-cluster.toml setup --variant gateway --plan
Scribe · local · setup plan · gateway variant · native runtime
  Definition  /srv/scribe/scribe-cluster.toml
  Paths below are relative to its directory.

  ENDPOINTS
    auth       tls://127.0.0.1:7443
    storage-a  tls://127.0.0.1:7447
    gateway    tls://127.0.0.1:7557

  CHECKS
    ok    definition    setup writes scribe-cluster.toml (gateway variant)
    ok    binaries      scribe-ctl, scribe-auth, scribe-server, scribe-gateway 0.1.0; setup records them as the first binary set in cluster-state/local/bin/0.1.0
    ok    permissions   0 existing secret files are private
    ok    cluster       stopped
    ok    ports         3 listen addresses are free
    ok    certificates  consistent
    ok    data roots    consistent

  STEPS
     1. Write the cluster definition scribe-cluster.toml and scribe-gateway.toml
        no definition exists yet
     2. Install scribe-ctl, scribe-auth, scribe-server, scribe-gateway 0.1.0 side by side in cluster-state/local/bin/0.1.0 (from releases/0.1.0)
        the cluster's executables, kept side by side so an upgrade can return to them
     3. Record 0.1.0 as the active set (bin/current)
        the runtime starts services from the set bin/current names
     4. Create the storage and gateway CA in pki/ca
        a server certificate needs this CA
     5. Create the auth CA in pki/auth-ca
        a server certificate needs this CA
     6. Issue auth's server certificate pki/auth/server.pem
        no certificate yet
     7. Issue storage-a's server certificate pki/storage-a/server.pem
        no certificate yet
     8. Issue gateway's server certificate pki/gateway/server.pem
        no certificate yet
     9. Initialize the gateway ledger data/gateway
        no gateway ledger yet
    10. Initialize auth data/auth with its pepper, revision witness and administrator credential
        no auth database yet
    11. Initialize storage storage-a in data/storage-a
        no repository store yet
    12. Bind storage storage-a to auth namespace studio
        the store is not bound to its auth namespace yet
    13. Register storage storage-a with auth
        auth must know the server before it issues its credential
    14. Ensure storage-a's server credential secrets/storage-a.credential
        issued when missing; an existing one is kept (`renew` replaces it before it expires)
    15. Ensure service account scribe-gateway reads and acts for users in studio
        the gateway reads inventory and acts for users in the namespaces it fronts
    16. Ensure scribe-gateway's credential secrets/gateway.credential
        issued when missing; an existing one is kept (`renew` replaces it before it expires)
    17. Ensure service account scribe-monitor reads in studio
        scribe-ctl reads storage inventory (df, repo list) as this account
    18. Ensure scribe-monitor's credential secrets/monitor.credential
        issued when missing; an existing one is kept (`renew` replaces it before it expires)
    19. Write the cluster summary CLUSTER-INFO.txt
        every file the cluster uses, its endpoints and the next commands

  NOTES
    - Auth runs temporarily for steps 13-18 and is stopped afterwards.
    - The CA private keys (ca-key.pem in pki/ca and pki/auth-ca) are the keys to the cluster: back them up offline and never share them. They are needed only to issue and renew server certificates and to rotate the CA (`renew --ca-dir DIR` signs with a key kept elsewhere).

  Plan only: nothing was written and no lock was taken. Run setup without --plan to apply it.

Docker runtime

With runtime = "docker" (setup --runtime docker) every service is a container of one Compose project that scribe-ctl generates from the definition and drives with the docker CLI (Docker Engine with Compose v2 and --wait; no Docker API client). The commands and plans are the same as natively; only where services run changes.

Images. setup --source DIR and upgrade --source DIR build the images from a Scribe checkout with the Dockerfiles in docker/ (and, with the web interface, every image apps/web/compose.yaml builds), before anything is planned, --plan included: every check runs against them, and building changes nothing in the cluster. They are tagged with this scribe-ctl's build (scribe/auth:BUILD, scribe/storage:BUILD, scribe/gateway:BUILD, scribe/web-SERVICE:BUILD; the build is passed as SCRIBE_BUILD_ID), and the checkout must be this scribe-ctl's version: a release's own controller runs it, so build scribe-ctl from the same checkout (set SCRIBE_BUILD_ID for a build of the same version) and run that; scripts/install.sh --setup and --upgrade do both, naming the build by the commit. A build id that cannot tag an image (+) is refused. Each image answers --build-info from a throwaway container (docker run --rm --network none), checked like executables: one release, built for the Docker engine's OS and architecture (docker version). The set is recorded as natively, <state_dir>/bin/<build>/SET.json (with the web interface's Compose file, seccomp profile and service list under web/), and bin/current selects the images the project runs.

The project. Before every start scribe-ctl writes <state_dir>/docker/: compose.yaml (JSON, which Compose reads as YAML), the gateway's configuration as its container reads it (gateway.toml), and with the web interface web.yaml (the active set's copy of apps/web/compose.yaml), its .env and the public CAs in trust/. Change the definition, not these files. The project is named scribe-CLUSTER-HASH, so two definitions with a cluster of the same name never share containers; where prints the docker compose command for it.

Containers see fixed paths, never the host's: a data root is /data, a service certificate /pki/server.pem and /pki/server-key.pem, a credential under /secrets/, auth's witness directory /witness. Host paths appear only as bind-mount sources (Windows' \\?\ prefixes removed), which must exist: Compose never creates one. Every container runs as the owner and group of the definition's directory (not on Windows, where the images' own user applies), with a read-only root, no capabilities and no-new-privileges, listens on its service's port on every interface of its own network namespace and publishes it on the definition's listen address only. Its health is its readiness file. A plaintext store therefore passes --allow-non-loopback inside its container: other containers of the project (the web interface) reach it on the project network, while the host sees only the published loopback address.

Containers reach each other by service name on the project network: tls://auth:7443, tls://storage-a:7447, tls://gateway:7557. Certificates name the service too, and the credentials of containers (each store's, the gateway's, the web reader's) carry auth's service address, while the administrator's and the inventory credential carry its published address, since scribe-ctl and people use them from the host. The gateway's container configuration names each store the cluster runs by its service name; every other setting is copied from the definition's gateway file.

Docker needs of the definition: a fixed port for every service; no [binaries]; and the revision witness in a directory of its own (the pepper may share it), because auth's container mounts that directory writable to replace the witness. Setup's docker layout uses secrets/auth/.

Tools and lifecycle. One-shot tools (init, auth-init, migrate) run as docker run --rm --network none of the set's image with exactly the mounts they need. Identities are provisioned from the host through auth's published port, as natively; setup starts auth alone (docker compose up -d --wait auth) while it needs it. up is docker compose up -d --wait --no-build --remove-orphans (the gateway, when require_healthy, is then waited for until it reports fresh inventory); down is docker compose stop (containers and their logs are kept, data is never touched); status reads docker compose ps; logs SERVICE is docker compose logs --tail N SERVICE for any service of the project. up --foreground starts detached as well and stops the project on Ctrl-C.

Web interface

setup --runtime docker --web adds a [web] table and runs the web interface (apps/web) in the cluster's project, with its own services and networks as apps/web/compose.yaml defines them. scribe-ctl adds only the wiring: its images; the cluster network for the services that reach Scribe (those on apps/web's upstream network); the public CAs (trust/ca.pem for storage and the gateway, trust/auth-ca.pem for auth); the reader credential's directory; the database's files in [web] data_dir (run as the directory's owner, so upgrades can back them up); and jwt_key and surreal_password, which setup writes (32 random bytes, base64, each atomically; a database whose password file is gone is refused, never given a new one). The .env points the web services at the gateway (or the first store) and auth by service name, and the Clone menu at the published addresses. With auth, setup creates the scribe-web service account (a namespace reader in every namespace of the cluster) and its credential. A new web database is prepared by the worker image's migrate.js, which the web services require before they start.

The web interface's own TLS certificate is yours: put fullchain.pem and key.pem in [web] tls_dir and set origin to the https:// address; the next start serves HTTPS on listen and redirects HTTP on redirect_port. Without them it serves plain HTTP; keep it on loopback. where says where the files go. See the web interface guide.

For a new gateway cluster with the web interface:

text

$ scribe-ctl --config ./scribe-cluster.toml setup --variant gateway --runtime docker --web --source /srv/scribe-src --plan
  Building scribe/auth:0.1.0 from /srv/scribe-src (the first build takes a while)…
  Building scribe/storage:0.1.0 from /srv/scribe-src (the first build takes a while)…
  Building scribe/gateway:0.1.0 from /srv/scribe-src (the first build takes a while)…
  Building the web interface's images (api, …) from /srv/scribe-src/apps/web…
Scribe · local · setup plan · gateway variant · docker runtime
  Definition  /srv/scribe/scribe-cluster.toml
  Paths below are relative to its directory.

  ENDPOINTS
    auth       tls://127.0.0.1:7443
    storage-a  tls://127.0.0.1:7447
    gateway    tls://127.0.0.1:7557
    web        http://localhost:8080

  CHECKS
    ok    definition    setup writes scribe-cluster.toml (gateway variant)
    ok    binaries      scribe-auth, scribe-server, scribe-gateway 0.1.0; setup records them as the first image set in cluster-state/local/bin/0.1.0
    ok    permissions   0 existing secret files are private
    FAIL  cluster       some of the cluster's containers run (storage-a, nginx, api, worker, database) but not auth, gateway; run `scribe-ctl --config /srv/scribe/scribe-cluster.toml down` first
    ok    certificates  consistent
    ok    data roots    consistent

  STEPS
     1. Write the cluster definition scribe-cluster.toml and scribe-gateway.toml
        no definition exists yet
     2. Record the images scribe/auth:0.1.0, scribe/storage:0.1.0, scribe/gateway:0.1.0 and the web interface's of 0.1.0 as a set in cluster-state/local/bin/0.1.0
        the cluster's executables, kept side by side so an upgrade can return to them
     3. Record 0.1.0 as the active set (bin/current)
        the runtime starts services from the set bin/current names
     4. Create the storage and gateway CA in pki/ca
        a server certificate needs this CA
     5. Create the auth CA in pki/auth-ca
        a server certificate needs this CA
     6. Issue auth's server certificate pki/auth/server.pem
        no certificate yet
     7. Issue storage-a's server certificate pki/storage-a/server.pem
        no certificate yet
     8. Issue gateway's server certificate pki/gateway/server.pem
        no certificate yet
     9. Initialize the gateway ledger data/gateway
        no gateway ledger yet
    10. Initialize auth data/auth with its pepper, revision witness and administrator credential
        no auth database yet
    11. Initialize storage storage-a in data/storage-a
        no repository store yet
    12. Bind storage storage-a to auth namespace studio
        the store is not bound to its auth namespace yet
    13. Create the web interface's missing secrets secrets/web/jwt.key and secrets/web/surreal.password
        the web services sign sessions and open their database with them
    14. Register storage storage-a with auth
        auth must know the server before it issues its credential
    15. Ensure storage-a's server credential secrets/storage-a.credential
        issued when missing; an existing one is kept (`renew` replaces it before it expires)
    16. Ensure service account scribe-gateway reads and acts for users in studio
        the gateway reads inventory and acts for users in the namespaces it fronts
    17. Ensure scribe-gateway's credential secrets/gateway.credential
        issued when missing; an existing one is kept (`renew` replaces it before it expires)
    18. Ensure service account scribe-monitor reads in studio
        scribe-ctl reads storage inventory (df, repo list) as this account
    19. Ensure scribe-monitor's credential secrets/monitor.credential
        issued when missing; an existing one is kept (`renew` replaces it before it expires)
    20. Ensure service account scribe-web reads in studio
        the web services read repositories as this account
    21. Ensure scribe-web's credential secrets/web-reader/scribe-web.credential
        issued when missing; an existing one is kept (`renew` replaces it before it expires)
    22. Prepare the web database data/web (migrate.js on the empty store)
        no web database yet
    23. Write the cluster summary CLUSTER-INFO.txt
        every file the cluster uses, its endpoints and the next commands

  NOTES
    - Auth runs temporarily for steps 14-21 and is stopped afterwards.
    - The CA private keys (ca-key.pem in pki/ca and pki/auth-ca) are the keys to the cluster: back them up offline and never share them. They are needed only to issue and renew server certificates and to rotate the CA (`renew --ca-dir DIR` signs with a key kept elsewhere).

  setup cannot run until every check passes.
  Plan only: nothing was written and no lock was taken. Run setup without --plan to apply it.

Remote operation

A cluster can live on another machine (a VM, a server) while scribe-ctl runs here. Give its definition a [remote] table, or pass --remote TARGET to any command; the definition file here stays the source of truth, and secrets stay there.

toml

[clusters.prod.remote]
host = "scribe-host"      # a host name, an address, or an alias from ~/.ssh/config
user = "deploy"           # optional: else the ssh config's, or yours
port = 22                 # optional
directory = "/srv/scribe" # holds the cluster there; absolute, or relative to the login's home

--remote takes [USER@]HOST:DIR, [USER@][IPV6]:DIR, ssh://[USER@]HOST[:PORT]/DIR (/~/DIR for one in the home directory), or a bare HOST whose directory comes from the definition's [remote]. It replaces the table for that command (user and port included). Host, user and directory are validated before anything runs: nothing that ssh could read as an option reaches it, and every word passed to the remote shell is quoted.

For each command, scribe-ctl here:

connects once with the system ssh in batch mode (-o BatchMode=yes, a 30 s connect timeout, keepalives every 15 s so a silently dropped connection ends within about a minute) and keeps that connection open as a master (ControlMaster, its socket in a private directory under /tmp) that every later ssh, tar and rsync of the command goes through: a command makes one connection however many steps it has. The master ends with the command, on success, error or Ctrl-C. Options on the command line win over the ssh config, so a ControlMaster set there does not interfere. On Windows, whose OpenSSH has no connection sharing, each step connects on its own. A first connection the remote times out, refuses or drops (a busy machine, a connection limit, a firewall throttling bursts) is tried four more times, after 10, 30, 60 and 90 s:
text
```
  ssh connect to scribe-host failed (ssh: connect to host scribe-host port 22: Operation timed out); retrying in 10 s (2/5)…
```
Authentication, host-key and name failures are not retried. Then it probes the remote: its OS and architecture (uname), rsync and docker, and the scribe-ctl builds installed in the cluster's directory;
pushes what the command needs: from the checkout of --source DIR, only what the image builds read, to upload/source (the Rust workspace, the Dockerfiles and apps/web, each without the paths its .dockerignore leaves out, and never .git, target*, node_modules, .env files, *.credential, *.pepper, secrets/, pki/ or cluster-state/; see what a push carries), and the executables of --binaries DIR to upload/binaries, which must run on the remote (its scribe-ctl answers --build-info there for the remote's OS and architecture). rsync 3 is used when both ends have it (only changes travel; everything else there is deleted); otherwise a tar stream through ssh, unpacked beside the previous upload and swapped in;
chooses the scribe-ctl that runs the command there, bin/<build>/scribe-ctl in the cluster's directory, installing it when it is missing:
- with --source, the one built from exactly the pushed checkout, in Docker there (docker/Dockerfile.ctl, exported as a file). Its build is <version>-src.<hash>: the first 12 hex digits of a BLAKE3 digest of the pushed files' paths and contents. Any change to the checkout builds a new scribe-ctl there, and nothing else does (modification times, the order files are listed in); the images it builds carry the same build. This needs docker on the remote;
- otherwise one rule, which doctor follows too:
  1. this scribe-ctl's exact build (same version, build and target) when it is there; on another OS or architecture only when its build id names a build (SCRIBE_BUILD_ID), since a build that is only the version may come from any source;
  2. else this build, installed: from the pushed --binaries when that is this release, or from this scribe-ctl's own directory (every executable of this release beside it) when both machines have the same OS and architecture;
  3. else the one bin/controller names when it is this version for the remote's machine: the newest installed there, by bootstrap or by the last command that installed one (Running the remote's scribe-ctl 0.1.0 (build 0.1.0-src.3fa9c2e1b7d0) (bin/controller): this 0.1.0 is built for macos/aarch64);
  4. else the one bin/current names, on the same terms;
  5. else this build after all, when it is there.
Otherwise it stops and names bootstrap. It installs into a new directory, moves an existing one aside, checks the installed scribe-ctl answers as that build, and only then names it in bin/controller and bin/current. Never system-wide. A remote whose bin/current names a newer version is refused: use that release here too. Builds of one version are one release, as everywhere: upgrading from one to another is allowed;
pushes the cluster's definition: the remote copy (DIR/<file name>) gets this file's version of the cluster without its [remote] table, other clusters there are kept, and a differing copy is kept as <file>.bak. A generated gateway configuration beside the definition goes along. Nothing is pushed for a cluster this file does not define yet (setup creates it there);
runs the same command there, cd DIR && exec bin/<build>/scribe-ctl --config <file> [--cluster NAME] [--json] … COMMAND … with the build chosen above, the pushed directories named as they are there. Discovery, planning, the command lock, the journal and execution all happen on the remote. Its stdout and stderr stream through unchanged (--json too), with a terminal when this one has one. An image build prints nothing until it ends, so while one runs the remote scribe-ctl says still building IMAGE (N min) once a minute. The remote command outlives its connection: when it drops (or this terminal closes) the scribe-ctl there carries on to the end, or through its rollback, with its output lost; status shows how it went, and running the command again resumes whatever it left pending. Ctrl-C asks it to stop at its next safe point, as locally: through the remote terminal, or without one (--json, redirected output) as kill -INT of the process id it announces, sent through the connection; a second Ctrl-C leaves it running and returns;
after a successful setup or add, brings the cluster's definition back into this file with its [remote] table (the previous file kept as <file>.bak), and after setup, add or upgrade brings the summary back as CLUSTER-INFO-<cluster>.txt beside it (its paths are the remote's). Nothing else comes back: CA keys, the pepper, credentials and data stay there.

The exit code is the remote scribe-ctl's; 255 means ssh itself failed. ssh never prompts: a host it cannot reach without a password or a host-key confirmation fails with what to do:

text

$ scribe-ctl --config ./scribe-cluster.toml --remote deploy@unreachable-vm:/srv/scribe status
  Checking deploy@unreachable-vm:/srv/scribe over ssh…
Scribe · status failed
  cannot reach deploy@unreachable-vm:/srv/scribe with ssh (ssh: Could not resolve hostname unreachable-vm: nodename nor servname provided, or not known). scribe-ctl runs the system ssh in batch mode: it uses your keys, agent and ~/.ssh/config and never asks for a password or to accept a host key. Make `ssh deploy@unreachable-vm true` work without a prompt (install a key with ssh-copy-id, connect once to accept the host key, or add a Host alias to ~/.ssh/config)

Setting up a cluster on a machine of the same OS and architecture as this one (here with a relative directory, in the login's home):

text

$ scribe-ctl --config ./scribe-cluster.toml --remote deploy@scribe-host:scribe setup --variant auth
  Checking deploy@scribe-host:scribe over ssh…
  Installing scribe-ctl 0.1.0 on the remote with this machine's scribe-ctl, scribe-auth, scribe-server, scribe-gateway…
  Running `setup` on deploy@scribe-host:scribe…
  local · [1/15] Write the cluster definition scribe-cluster.toml…
  …
Scribe · local · Setup complete
  wrote scribe-cluster.toml
  installed 0.1.0 in cluster-state/local/bin/0.1.0
  …
  wrote CLUSTER-INFO.txt

  Every file, endpoint and next command: /home/deploy/scribe/CLUSTER-INFO.txt (or `where`).
  Next: scribe-ctl --config /home/deploy/scribe/scribe-cluster.toml up
  Wrote cluster local into /home/me/ops/scribe-cluster.toml with its [remote] table…
  Brought the cluster summary back as /home/me/ops/CLUSTER-INFO-local.txt (paths in it are on the remote)…

The remote copy of the definition has this file's name. From then on the table is in this file and every command goes there without --remote:

text

$ scribe-ctl --config ./scribe-cluster.toml up
  Checking deploy@scribe-host:scribe over ssh…
  Running `up` on deploy@scribe-host:scribe…
  local · Starting cluster…
Scribe · local · running
  SERVICE    STATE        PID  ENDPOINT
  auth       running    75409  tls://127.0.0.1:7443
  storage-a  running    75421  tls://127.0.0.1:7447

  Logs  /home/deploy/scribe/cluster-state/local/logs
  Binaries  0.1.0 · /home/deploy/scribe/cluster-state/local/bin/0.1.0
$ scribe-ctl --config ./scribe-cluster.toml logs nonexistent; echo "exit $?"
  Checking deploy@scribe-host:scribe over ssh…
  Running `logs` on deploy@scribe-host:scribe…
Scribe · logs failed
  unknown service
exit 1

Output of the remote command names the remote's paths (its Next: line is the remote's command line too). Services listen where the definition says, on the remote: setup's layouts listen on loopback, which is the remote's own. For clients elsewhere, write the definition here first (scribe-ctl example, or a setup --plan to start from), with the listen addresses and certificate names ([tls] dns/ip) clients use and the [remote] table, and run setup: it pushes that definition and sets it up as written. --plan runs remotely like any command; it pushes the definition and may install scribe-ctl in bin/ first, but changes nothing of the cluster. With several clusters in this file and no --cluster, status and where cover the local ones and name each remote one with the command that asks it. Every command works remotely except example (local) and the hidden supervisor; for a shell there, see ssh.

On the remote, the cluster's directory holds the definition's copy, bin/<build>/ (a scribe-ctl per release that ran there, with the service executables it was installed with: native setup records those, so a native cluster's first binary set comes from there), bin/controller (the newest scribe-ctl installed there), bin/current (the one the last command that installed one ran with; for a --source setup or upgrade, the build its images carry), upload/source and upload/binaries, upload/controller/ (what bootstrap pushed), and whatever the definition names (data, secrets, pki, state). These bin/ files are the remote directory's, not the cluster's active set in <state_dir>/bin/current. Remove old bin/<build> directories by hand.

add auth

Put auth in front of a storage-only cluster: CAs, certificates, the auth database, and TLS with sign-in on every store. Repositories are kept. The cluster must be stopped

text

scribe-ctl add auth [OPTIONS]

Argument or option	Meaning
`--namespace <NAME>`	Auth namespace every store joins [default: studio]
`--listen <ADDR>`	Where auth listens [default: 127.0.0.1 on 7443, moved by as much as the first store's port is from 7447]
`--plan`	Show the plan with the definition diff and run every read-only check, without writing anything or taking a lock

add auth converts a storage-only cluster (the storage variant, or any definition of plaintext stores with no auth and no gateway) into the auth variant. It rewrites the definition canonically, keeping the previous file as <file>.bak: an [auth] table (data, pepper, revision witness, administrator credential and certificate under data/, secrets/ and pki/), a [certificates] table when there is none (an existing one is kept as written), and on every store auth = true, the namespace, its credential, the scribe-monitor inventory credential and its certificate. A store keeps its address; only the scheme changes from tcp:// to tls://. In a file that defines several clusters the new paths start with the cluster's name.

Then it provisions exactly what setup would for that definition: both CAs if missing, the auth and store certificates, the auth database, each store bound to its namespace (scribe-server auth-init) in place, its registration and server credential, and the inventory account. Data roots are never initialized again or deleted: repositories and their history stay, and the storage data needs no copy. Once bound, a store refuses to start without auth, so the cluster must be stopped for the whole change; while it runs, the plan fails its cluster check and nothing is changed. Run down, then add auth, then up.

What changes for people is listed under AFTERWARDS: clients switch to tls:// with --tls-ca pki/ca/ca.pem and log in through auth with --auth-ca pki/auth-ca/ca.pem; nobody has access until an administrator adds people and grants (the Next commands in CLUSTER-INFO.txt); and a workspace created before auth has no signed-in owner, so it still syncs but cannot submit: create a new workspace after logging in.

add auth is resumable like setup, with the same journal (operation add auth), set-aside rule and refusal of state it did not make. A rerun after it completed, or on a cluster that already has auth, changes nothing and says nothing to add; one whose auth is incomplete without an interrupted add is refused with a pointer to setup. --namespace and --listen are refused once auth exists, and so is a custom topology (for example plaintext and TLS stores side by side). A listen port another service of the file uses is refused. For a storage cluster set up with default ports:

text

$ scribe-ctl --config ./scribe-cluster.toml add auth --plan
Scribe · local · add auth plan · auth variant · native runtime
  Definition  /srv/scribe/scribe-cluster.toml
  Paths below are relative to its directory.

  ENDPOINTS
    auth       tls://127.0.0.1:7443
    storage-a  tls://127.0.0.1:7447

  CHECKS
    ok    definition    add auth rewrites scribe-cluster.toml with auth
    ok    binaries      scribe-ctl, scribe-auth, scribe-server 0.1.0, the active set in cluster-state/local/bin/0.1.0
    ok    permissions   0 existing secret files are private
    ok    cluster       stopped
    ok    ports         2 listen addresses are free
    ok    certificates  consistent
    ok    data roots    consistent

  DEFINITION CHANGES
    @ scribe-cluster.toml
      [clusters.local]
      # What setup builds: storage, auth or gateway. Setup refuses services that do not match it.
    - variant = "storage"
    + variant = "auth"
      # Supervisor state, service logs and setup's journal. Never share it between clusters.
      state_dir = "cluster-state/local"
    …
      startup_timeout_secs = 30

    + # The two server-certificate CAs.
    + [clusters.local.certificates]
    + # Setup may create missing CAs and server certificates. Once they exist, move the
    + # CA private keys (ca-key.pem) offline and set this to false.
    + generate_missing = true
    + # Signs storage and gateway certificates.
    + ca_dir = "pki/ca"
    + # Signs only auth's certificate, so no storage or gateway key can pass as auth.
    + auth_ca_dir = "pki/auth-ca"
    +
    + # Accounts, credentials and access decisions. TLS only.
    + [clusters.local.auth]
    + # A fixed port: credentials name this address.
    + listen = "127.0.0.1:7443"
    + data_dir = "data/auth"
    + # Written by setup and needed by every auth start. Back it up apart from the data.
    + pepper_file = "secrets/auth.pepper"
    + # Revision witness: keep it outside the data directory and its backups.
    + revision_file = "secrets/auth.revision"
    + # The bootstrap administrator's credential, written by setup. Keep it private.
    + admin_credential = "secrets/admin.credential"
    +
    + # Server certificate and key; setup issues them when generate_missing is on.
    + [clusters.local.auth.tls]
    + cert = "pki/auth/server.pem"
    + # Private key: never share it.
    + key = "pki/auth/server-key.pem"
    +
      # A storage server.
      [[clusters.local.storage]]
    …
      data_dir = "data/storage-a"
      # Clients sign in through auth; requires TLS.
    - auth = false
    + auth = true
    + # The auth namespace this server's repositories belong to.
    + namespace = "studio"
    + # This server's own credential, issued by setup. Keep it private.
    + credential = "secrets/storage-a.credential"
    + # A namespace reader's credential; scribe-ctl uses it for df, repo list and status --storage.
    + monitor_credential = "secrets/monitor.credential"
    +
    + # Server certificate and key; setup issues them when generate_missing is on.
    + [clusters.local.storage.tls]
    + cert = "pki/storage-a/server.pem"
    + # Private key: never share it.
    + key = "pki/storage-a/server-key.pem"

  STEPS
     1. Rewrite the definition scribe-cluster.toml with auth, keeping the previous one as scribe-cluster.toml.bak
        the definition has no auth yet; it is rewritten canonically
     2. Create the storage and gateway CA in pki/ca
        a server certificate needs this CA
     3. Create the auth CA in pki/auth-ca
        a server certificate needs this CA
     4. Issue auth's server certificate pki/auth/server.pem
        no certificate yet
     5. Issue storage-a's server certificate pki/storage-a/server.pem
        no certificate yet
     6. Initialize auth data/auth with its pepper, revision witness and administrator credential
        no auth database yet
     7. Bind storage storage-a to auth namespace studio
        the store is not bound to its auth namespace yet
     8. Register storage storage-a with auth
        auth must know the server before it issues its credential
     9. Ensure storage-a's server credential secrets/storage-a.credential
        issued when missing; an existing one is kept (`renew` replaces it before it expires)
    10. Ensure service account scribe-monitor reads in studio
        scribe-ctl reads storage inventory (df, repo list) as this account
    11. Ensure scribe-monitor's credential secrets/monitor.credential
        issued when missing; an existing one is kept (`renew` replaces it before it expires)
    12. Write the cluster summary CLUSTER-INFO.txt
        every file the cluster uses, its endpoints and the next commands

  NOTES
    - The definition file is rewritten in canonical form; the previous one is kept as scribe-cluster.toml.bak.
    - Auth runs temporarily for steps 8-11 and is stopped afterwards.
    - The CA private keys (ca-key.pem in pki/ca and pki/auth-ca) are the keys to the cluster: back them up offline and never share them. They are needed only to issue and renew server certificates and to rotate the CA (`renew --ca-dir DIR` signs with a key kept elsewhere).
    - Storage data and repositories are kept: every store is bound to namespace studio in place, never re-initialized. Once bound, a store no longer starts without auth.

  AFTERWARDS
    - Start it with `scribe-ctl --config /srv/scribe/scribe-cluster.toml up`: auth first, then storage.
    - Clients switch from tcp:// to tls://127.0.0.1:7447 (trusting pki/ca/ca.pem with --tls-ca) and log in through auth at tls://127.0.0.1:7443 (trusting pki/auth-ca/ca.pem with --auth-ca); plaintext connections are refused from now on.
    - Nobody has access yet: add people and grants with scribe-auth and the administrator credential secrets/admin.credential (the Next commands in CLUSTER-INFO.txt).
    - Workspaces created before auth have no signed-in owner: they still sync, but submitting needs a new workspace (scribe workspace create) made after logging in; bring unsubmitted work over by hand.
    - Give clients pki/ca/ca.pem and pki/auth-ca/ca.pem; keep the CA private keys (ca-key.pem beside them) offline.

  Plan only: nothing was written and no lock was taken. Run add auth without --plan to apply it.

add gateway

Put a TLS gateway in front of an auth cluster's stores, with the scribe-gateway service account and its credential. The cluster must be stopped

text

scribe-ctl add gateway [OPTIONS]

Argument or option	Meaning
`--listen <ADDR>`	Where the gateway listens [default: 127.0.0.1 on 7557, moved by as much as auth's port is from 7443]
`--plan`	Show the plan with the definition diff and run every read-only check, without writing anything or taking a lock

add gateway turns the auth variant (auth and TLS stores, no gateway) into the gateway variant. It writes scribe-gateway.toml beside the definition (one [[instances]] entry per store, in the store's namespace, trusting the cluster CA) and rewrites the definition with a [gateway] table, keeping the previous file as <file>.bak; --plan shows both. It then issues the gateway's certificate from the cluster CA, initializes the gateway ledger, and ensures the scribe-gateway service account with namespace reader and delegate grants in every namespace it fronts and its credential. Repositories created directly on a store before the gateway existed are imported by its first inventory poll and keep their IDs; through the gateway their names carry the namespace (studio/game).

The cluster must be stopped. up then starts auth, then storage, then the gateway, and down drains them in reverse. A client logs in once more with the gateway as --server: a saved session belongs to the server it was made for.

On a storage-only cluster add gateway is refused (run add auth first), as it is when a store listens on port 0 (the gateway needs its fixed address). On a cluster that already has a gateway it changes nothing and says nothing to add. Resume and refusal work as for add auth.

upgrade

Upgrade the cluster to the release in a directory of executables: sanity checks, backups of the roots a migration changes, explicit migrations, a health check, and rollback on failure. Resumable; one at a time

text

scribe-ctl upgrade [OPTIONS] --binaries <DIR>
scribe-ctl upgrade [OPTIONS] --source <DIR>

Argument or option	Meaning
`--binaries <DIR>`	Native: directory with the new release's executables: scribe-ctl and those the cluster runs (scribe-auth, scribe-server, scribe-gateway); a scribe client beside them is reported. A newer scribe-ctl there takes the upgrade over
`--source <DIR>`	Docker: the Scribe checkout of this scribe-ctl's release to build the new images from (and the web interface's, with [web]); built before planning, also with --plan
`--plan`	Show the plan and run every read-only check and migration dry run, without writing anything, taking a lock or touching a service

upgrade moves a set-up cluster from its active set to the release in DIR: natively a directory of executables (--binaries), under docker a Scribe checkout whose images it builds first (--source, see Docker runtime); the other option is refused. It needs a recorded set (see setup): a cluster without bin/current is refused with a pointer to setup.

Which scribe-ctl runs it. Only a release's own controller knows that release's migrations. Under docker the images are built as the running scribe-ctl's own release, so there is nothing to hand over: build scribe-ctl from the checkout and run it. When DIR holds a scribe-ctl of a newer version, or of the same version and another build, the running scribe-ctl hands the whole command over to it: it runs DIR/scribe-ctl with the same options (--plan and --json included) and exits with its exit code. An older scribe-ctl in DIR is not handed anything; the plan refuses it as a downgrade. The plan's controller check names who planned it.

Checks. Every check below runs for real under --plan too, and any failure refuses the whole upgrade before anything changes:

new set: DIR holds scribe-ctl and every executable the cluster runs; each answers --build-info (see JSON output), names itself correctly and is built for this machine's OS and architecture; all report one version and build. A scribe client beside them is reported, never required or installed.
release: the new version is not older than the active one (SemVer order); the same version with another build is allowed. Upgrading to the active build changes nothing and says so.
formats: which persistent formats and protocols change between the sets, from the executables' build info. A format going back refuses the upgrade.
<component> data: each data root's recorded formats, read without opening or locking it (the storage data-root descriptor, the SCRIBE_AUTH marker, the SCRIBE_GATEWAY marker), against what the new executables read and write. Data newer than they support is refused.
permissions: existing secret files, and auth's data root, grant nothing to other accounts. Auth refuses to start (or migrate) from a root others can enter, so a mode like 0755 fails the plan with the chmod 700 <root> that fixes it, before anything is drained.
cluster, journal: a busy cluster, or another command's interrupted run, is refused.

Migrations. In dependency order, each after a backup: auth with the new scribe-auth migrate (which rewrites the revision witness), each store with the new scribe-server migrate, and under docker the web database with the new worker image's migrate.js. While the cluster is stopped, the plan runs scribe-auth migrate --dry-run and scribe-server migrate --dry-run with the new executables and migrates exactly the roots they report pending (a refused dry run refuses the upgrade). While it runs, a root's data-root lock rules a dry run out, so a root is migrated when its formats change between the sets or it records an older format than the new executable; each migrate checks everything before it writes. Two migrations are refused with what to do by hand: auth's schema 3 to 4, which replaces every credential (the administrator's, every server's and service account's), and any of the gateway ledger, since scribe-gateway has no migrate command.

The web database migrates with every web release, since migrate.js records the release's schema even when no migration is pending. While the old release serves, the plan runs the new worker's migrate.js --dry-run against the running database, so data the new release would refuse stops the upgrade before anything stops; a stopped database is dry-run by the migrate step itself, first.

Auth's revision witness. Auth refuses to start on a database behind its witness, the file that detects a restored old copy. The witness stays outside auth's data root and its backups, so the upgrade records it (a line of incarnation and revision) in its journal when it backs auth up, after the drain, before copying. A rollback that restores auth writes the recorded witness back with the database, so the two agree again; the journal, and with it the only copy, is removed when the upgrade completes, so no backup ever carries a witness that would let an old copy pass. A change made through the new release's auth between its start and a rollback is undone with the rest.

Steps. Install the new set side by side in <state_dir>/bin/<build>/ (each executable copied, checked to answer as its source did, then SET.json; under docker the images recorded), while the cluster still serves; drain it if it runs; back up each root that will be migrated; switch bin/current; migrate; start every service from the new set and wait until each is ready (the health check, as up --daemon); stop again if the cluster was stopped before; rewrite CLUSTER-INFO.txt. For a running gateway cluster with nothing to migrate:

text

$ scribe-ctl --config ./scribe-cluster.toml upgrade --binaries releases/next --plan
  Handing the upgrade over to /srv/scribe/releases/next/scribe-ctl of 0.1.0 (build next), which knows its release's migrations…
Scribe · local · upgrade plan · gateway variant · native runtime
  Definition  /srv/scribe/scribe-cluster.toml
  Paths below are relative to its directory.

  ENDPOINTS
    auth       tls://127.0.0.1:7443
    storage-a  tls://127.0.0.1:7447
    gateway    tls://127.0.0.1:7557

  CHECKS
    ok    release         0.1.0 → 0.1.0 (build next): the same version, another build
    ok    installed set   0.1.0 in cluster-state/local/bin/0.1.0
    ok    new set         releases/next: scribe-ctl, scribe-auth, scribe-server, scribe-gateway 0.1.0 (build next) for macos/aarch64, and the scribe client (not installed)
    ok    controller      scribe-ctl 0.1.0 (build next) from releases/next (handed over by scribe-ctl 0.1.0)
    ok    formats         unchanged: every executable reads and writes what the active set does
    ok    cluster         running; drained after the new set is installed
    ok    permissions     10 existing secret files and auth's data root are private
    ok    auth data       auth_schema 4; no migration: its formats do not change (no dry run: auth serves it and holds its data-root lock)
    ok    storage-a data  catalog_schema 1, envelope 1, object_store 1, repository_schema 3; no migration: its formats do not change (no dry run: storage-a serves it and holds its data-root lock)
    ok    gateway data    route_store_schema 2; no migration

  STEPS
     1. Install scribe-ctl, scribe-auth, scribe-server, scribe-gateway 0.1.0 (build next) side by side in cluster-state/local/bin/next (from releases/next)
        side by side with the active set, before anything stops
     2. Drain the running cluster
        roots are backed up, migrated and switched only while nothing holds them
     3. Switch the active set from 0.1.0 to next (bin/current)
        services start from the set bin/current names
     4. Start the cluster with next and wait until every service is ready
        the health check: every service must become ready
     5. Write the cluster summary CLUSTER-INFO.txt
        the summary names the active set

  ROLLBACK (if step 2 or a later one fails)
     6. Drain whatever of the cluster runs
        nothing may hold a root while it is restored
     7. Switch the active set from next to 0.1.0 (bin/current)
        back to the executables that served before
     8. Start the cluster again with 0.1.0
        it was running before the upgrade
     9. Write the cluster summary CLUSTER-INFO.txt
        the summary names the active set

  NOTES
    - The scribe-ctl in releases/next decides this upgrade: a release's own scribe-ctl runs its upgrade, because only it knows that release's migrations. An older scribe-ctl given a newer one in --binaries hands the upgrade over to it.
    - No data root changes, so nothing is backed up.
    - If step 2 or a later one fails, the rollback below runs and the cluster returns to 0.1.0.
    - A failure before step 2 changes nothing: 0.1.0 stays active and the cluster keeps serving; run upgrade again to continue.
    - Interrupted (a crash, a power loss)? Run the same command again: it resumes from cluster-state/local/journal.json, or finishes a rollback.

  AFTERWARDS
    - Run cluster-state/local/bin/next/scribe-ctl from now on (or copy it where you keep scribe-ctl): it is this release's controller.

  Plan only: nothing was written and no lock was taken. Run upgrade without --plan to apply it.

For the same cluster, stopped, with a pending migration of its store:

text

$ scribe-ctl --config ./scribe-cluster.toml upgrade --binaries releases/next --plan
  Handing the upgrade over to /srv/scribe/releases/next/scribe-ctl of 0.1.0 (build next), which knows its release's migrations…
  …
  CHECKS
    ok    release         0.1.0 → 0.1.0 (build next): the same version, another build
    ok    installed set   0.1.0 in cluster-state/local/bin/0.1.0
    ok    new set         releases/next: scribe-ctl, scribe-auth, scribe-server, scribe-gateway 0.1.0 (build next) for macos/aarch64, and the scribe client (not installed)
    ok    controller      scribe-ctl 0.1.0 (build next) from releases/next (handed over by scribe-ctl 0.1.0)
    ok    formats         unchanged: every executable reads and writes what the active set does
    ok    cluster         stopped; started for the health check, then stopped again
    ok    permissions     10 existing secret files and auth's data root are private
    ok    auth data       auth_schema 4; dry run: nothing to migrate
    ok    storage-a data  catalog_schema 1, envelope 1, object_store 1, repository_schema 2; dry run: the data-root descriptor is upgraded; backed up, then migrated
    ok    gateway data    route_store_schema 2; no migration
    ok    free space      177.33 KiB to copy to cluster-state/local/backups/20260927T103411654Z, 424.11 GiB available

  STEPS
     1. Install scribe-ctl, scribe-auth, scribe-server, scribe-gateway 0.1.0 (build next) side by side in cluster-state/local/bin/next (from releases/next)
        side by side with the active set, before anything stops
     2. Back up storage storage-a (data/storage-a) to cluster-state/local/backups/20260927T103411654Z/storage-a
        the migration changes it; a rollback restores this copy
     3. Switch the active set from 0.1.0 to next (bin/current)
        services start from the set bin/current names
     4. Migrate storage storage-a (data/storage-a) with the new scribe-server migrate
        explicit, with the new executables; reruns resume where it stopped
     5. Start the cluster with next and wait until every service is ready
        the health check: every service must become ready
     6. Stop the cluster again, as it was before
        it was stopped before the upgrade
     7. Write the cluster summary CLUSTER-INFO.txt
        the summary names the active set

  ROLLBACK (if step 3 or a later one fails)
     8. Drain whatever of the cluster runs
        nothing may hold a root while it is restored
     9. Restore storage storage-a from cluster-state/local/backups/20260927T103411654Z/storage-a if it was migrated, keeping the migrated root as data/storage-a.upgraded-20260927T103411654Z
        the previous release cannot read what the migration wrote
    10. Switch the active set from next to 0.1.0 (bin/current)
        back to the executables that served before
    11. Write the cluster summary CLUSTER-INFO.txt
        the summary names the active set
  …

Backups. Each root a migration will change (auth, a store, the web database) is copied, stopped, to <state_dir>/backups/<UTC stamp>/<name>/ (auth, the store's name, web): directories private (0700), files keeping their modes, every file and directory flushed, and complete only once <name>.json (its manifest) is written. The manifest lists every directory and every file with its size and BLAKE3 digest; the copy is read back and checked against them before the manifest is written. The web database backup is a cold copy of its SurrealDB files, consistent because the database is stopped. Roots that do not change are not copied. The plan's free space check measures the roots to copy and refuses the upgrade unless the filesystem holding <state_dir>/backups has room for them plus a reserve of a tenth of their size, at least 1 GiB (a full filesystem would stop the services on it too). Backups and earlier binary sets are kept; scribe-ctl never deletes either. where lists them.

Rollback. A failure once a running cluster is drained (for a stopped cluster, from the switch on) rolls back: drain whatever runs, restore each root whose migration started (auth with its witness; the migrated root is moved aside to <root>.upgraded-<stamp> and kept; the backup is checked against its manifest, copied beside it, checked again and renamed into place), switch bin/current back, start the previous set if the cluster ran before, and report. So a backup that fails after the drain restarts the old set rather than leaving the cluster down. The new set stays installed. A failure earlier (the install) changes nothing and leaves the previous set serving. For example, a release whose storage server cannot start:

text

$ scribe-ctl --config ./scribe-cluster.toml upgrade --binaries releases/next
  Handing the upgrade over to /srv/scribe/releases/next/scribe-ctl of 0.1.0 (build next), which knows its release's migrations…
  local · [1/5] Install scribe-ctl, scribe-auth, scribe-server, scribe-gateway 0.1.0 (build next) side by side in cluster-state/local/bin/next (from releases/next)…
  local · [2/5] Drain the running cluster…
  local · [3/5] Switch the active set from 0.1.0 to next (bin/current)…
  local · [4/5] Start the cluster with next and wait until every service is ready…
  local · [6/9] Drain whatever of the cluster runs…
  local · [7/9] Switch the active set from next to 0.1.0 (bin/current)…
  local · [8/9] Start the cluster again with 0.1.0…
  local · [9/9] Write the cluster summary CLUSTER-INFO.txt…
Scribe · local · Upgrade to next rolled back
  step 4 (Start the cluster with next and wait until every service is ready) failed: supervisor exited (exit status: 1): storage-a failed to start; inspect /srv/scribe/cluster-state/local/logs/storage-a.log; its log is /srv/scribe/cluster-state/local/logs/supervisor.log
  installed 0.1.0 (build next) in cluster-state/local/bin/next
  drained the cluster
  switched the active set from 0.1.0 to next

  ROLLED BACK
    nothing was running
    switched the active set from next to 0.1.0
    every service is ready, running 0.1.0
    wrote CLUSTER-INFO.txt

  The active set is 0.1.0 again, and the cluster runs.
  The new set stays installed; fix the cause, then run upgrade again.

Docker. The same plan, with images: for a running storage cluster with the web interface, upgraded with the next release's scribe-ctl (build next):

text

$ scribe-ctl --config ./scribe-cluster.toml upgrade --source /srv/scribe-src --plan
  local · Building scribe/auth:next from /srv/scribe-src (the first build takes a while)…
  local · Building scribe/storage:next from /srv/scribe-src (the first build takes a while)…
  local · Building scribe/gateway:next from /srv/scribe-src (the first build takes a while)…
  local · Building the web interface's images (api, …) from /srv/scribe-src/apps/web…
Scribe · local · upgrade plan · storage variant · docker runtime
  Definition  /srv/scribe/scribe-cluster.toml
  Paths below are relative to its directory.

  ENDPOINTS
    storage-a  tcp://127.0.0.1:7447
    web        http://localhost:8080

  CHECKS
    ok    release        0.1.0 → 0.1.0 (build next): the same version, another build
    ok    installed set  0.1.0 in cluster-state/local/bin/0.1.0
    ok    new set        /srv/scribe-src: scribe-auth, scribe-server, scribe-gateway 0.1.0 (build next) for linux/x86_64
    ok    controller     scribe-ctl 0.1.0 (build next)
    ok    formats        unchanged: every executable reads and writes what the active set does
    ok    cluster        running; drained after the new set is installed
    …
    ok    web data       dry run: no migration pending; data/web is backed up, then the new worker's migrate.js runs (its dry run first)
    …

  STEPS
     1. Record the images scribe/auth:next, scribe/storage:next, scribe/gateway:next and the web interface's of 0.1.0 (build next) as a set in cluster-state/local/bin/next
        side by side with the active set, before anything stops
     2. Drain the running cluster
        roots are backed up, migrated and switched only while nothing holds them
     3. Back up the web database (data/web) to cluster-state/local/backups/20260927T112353229Z/web
        the migration changes it; a rollback restores this copy
     4. Switch the active set from 0.1.0 to next (bin/current)
        services start from the set bin/current names
     5. Migrate the web database (data/web) with the new worker's migrate.js, after its dry run
        explicit, with the new executables; reruns resume where it stopped
     6. Start the cluster with next and wait until every service is ready
        the health check: every service must become ready
     7. Write the cluster summary CLUSTER-INFO.txt
        the summary names the active set

  ROLLBACK (if step 2 or a later one fails)
     8. Drain whatever of the cluster runs
        nothing may hold a root while it is restored
     9. Restore the web database from cluster-state/local/backups/20260927T112353229Z/web if it was migrated, keeping the migrated root as data/web.upgraded-20260927T112353229Z
        the previous release cannot read what the migration wrote
    10. Switch the active set from next to 0.1.0 (bin/current)
        back to the executables that served before
    11. Start the cluster again with 0.1.0
        it was running before the upgrade
    12. Write the cluster summary CLUSTER-INFO.txt
        the summary names the active set

  NOTES
    - This scribe-ctl decides this upgrade: it built the images from /srv/scribe-src as its own release, because only a release's own scribe-ctl knows its migrations.
    - Backups go to cluster-state/local/backups/20260927T112353229Z; they and the previous set (cluster-state/local/bin/0.1.0) are kept. scribe-ctl never deletes either; remove them yourself once the new release has proven itself.
    - If step 2 or a later one fails, the rollback below runs and the cluster returns to 0.1.0 with its data as backed up.
    - A failure before step 2 changes nothing: 0.1.0 stays active and the cluster keeps serving; run upgrade again to continue.
    - Interrupted (a crash, a power loss)? Run the same command again: it resumes from cluster-state/local/journal.json, or finishes a rollback.

  AFTERWARDS
    - Keep running this scribe-ctl (built from the same checkout as the images): it is this release's controller.

  Plan only: nothing was written and no lock was taken. Run upgrade without --plan to apply it.

Resume and lock. The upgrade records its decisions (sets, backup directory, stores to migrate, whether the cluster ran) and every step in the journal before it runs. After a crash or power loss, run the same command again: it continues from the journal, moving a partially copied set or backup aside as <name>.interrupted-<run>. A rollback that stopped is finished by the next run from the journal alone. The upgrade holds the command lock for its whole run, so a second upgrade, up, down, setup or add is refused meanwhile; --plan takes no lock.

Exit 0 means upgraded (or nothing to upgrade), 1 means refused, failed or rolled back. With --json, a completed upgrade reports plan and results; a rolled back one reports ok: false, rolled_back: true, failure, results (the steps before the failure), rollback, active_set and backups.

prune

Remove what old releases left behind, and only what scribe-ctl made: binary sets other than the active one and the --keep newest before it (for a rollback), the images tagged with those sets' builds that no container uses, the scribe-ctl builds a remote command installed for them, and upgrade backups older than the --keep newest. Docker's build cache, other images, volumes and containers are never touched. Never run by renew; you run it when you choose

text

scribe-ctl prune [OPTIONS]

Argument or option	Meaning
`--plan`	Show what would be removed, and why each is old, without removing anything or taking a lock
`--keep <N>`	How many sets before the active one stay, and how many of the newest upgrade backups [default: 1]

Sets are ordered by when they were installed. An image a container uses (running or stopped) stays, and the plan names it. A directory in the cluster's bin/ counts as a scribe-ctl build only when its name is a build id and it holds scribe-ctl; anything else there is left alone. The removals are independent and each is idempotent, so an interrupted prune is simply run again. Exit 0 when pruned or nothing is old, 1 when refused (no active set is recorded).

renew

Replace everything the cluster renews that expires within 30 days: service credentials and the administrator credential (issued, put in place, verified, then the old one revoked) and server certificates (reissued for the key they have by the CA that signed them, then a TLS handshake proves each); and retire the old CA of every rotation whose overlap has ended. Holders restart one at a time (or a stopped cluster starts); a failed verification puts the old files back. The web interface's own certificate stays yours. Non-interactive, for schedulers: exit 0 when renewed or nothing is due, 1 with the old files kept. With --rotate-ca it starts a CA rotation instead (never done by a plain renew); with --rekey it replaces servers' private keys instead. Resumable; one at a time

text

scribe-ctl renew [OPTIONS]

Argument or option	Meaning
`--plan`	Show what would be renewed, rotated or retired and when each expires, running every read-only check, without writing anything, taking a lock or touching a service
`--ca-dir <DIR>`	The storage and gateway CA's directory (its ca.pem and ca-key.pem) when its key is not kept beside the cluster. With --remote the key stays on this machine: certificates are signed here and only they travel to the cluster
`--auth-ca-dir <DIR>`	The auth CA's directory (its ca.pem and ca-key.pem) when its key is not kept beside the cluster; as --ca-dir
`--rotate-ca <CA>`	Start a CA rotation (a plain renew never does). storage or auth, CAs clients trust: a new CA and a cross certificate the old one signs for it; every server certificate of that CA is reissued by the new CA for its key and followed by the cross certificate, so clients that trust only the old CA keep connecting and learn the new one during the overlap, after which a plain renew retires the old CA. credential or access, auth's internal client CAs (only Scribe's services trust them): a successor is added while the cluster drains for a moment
`--retire-ca <CA>`	Finish a rotation now. storage or auth: before its overlap ends (a plain renew retires it at the end), so clients that have not connected since it started will see "CA changed" and trust the new CA by hand. credential or access: retire auth's oldest internal CA, which auth refuses while anything it signed is in use
`--rekey [<SERVICE>...]`	Replace TLS servers' private keys (a plain renew never does): each named service (comma-separated or repeated; every TLS server of the cluster when none is named) gets a new key, generated where it runs, and a certificate for it from the CA that signs it now, for the same names. The old key and certificate are kept until a TLS handshake proves the new pair, and go back if it fails. Clients trust the CA, not the key: nothing changes for them. For a key suspected leaked, or to move servers to a post-quantum key
`--overlap-days <DAYS>`	With --rotate-ca storage\|auth: the days both CAs are trusted, which is how long clients have to connect and learn the new CA [default: 90]

<CA> is storage (the storage and gateway CA, certificates.ca_dir), auth (the auth CA, certificates.auth_ca_dir), credential or access (auth's internal client CAs).

What renew owns

Credentials: each authenticated store's server credential, the gateway's (scribe-gateway), the scribe-monitor inventory credential, under docker with the web interface the scribe-web reader's, and the administrator credential the definition names ([auth] admin_credential; the bootstrap one setup writes never expires, an operator's does). Each file's expiry is read from the file itself; one within 30 days, or expired, is due.
Server certificates of auth, every store and the gateway, due the same way. The web interface's fullchain.pem is yours (web interface).
CA rotations whose overlap has ended: their old CA is retired (CA rotation).

With nothing due, renew exits 0 and writes nothing, not even a lock:

text

$ scribe-ctl --config ./scribe-cluster.toml renew
Scribe · local · Unchanged: nothing to renew: 4 credentials are valid for 30 more days; the first to expire is the storage-a server credential on 2027-09-27
  3 server certificates are valid for 30 more days; the first to expire is auth's on 2027-09-27

Credentials

For the due credentials it runs, in order:

auth issues each replacement into <file>.next (365 days; an administrator credential 3650) for the same holder; the old one stays valid;
each is put in place: the current file is kept as <file>.previous (durably), then the new one is renamed over it;
the services that read their credential only when they start restart: the stores, then the gateway, one at a time while the others keep serving (natively the supervisor stops and starts each in place and waits until it is ready; under docker its container is recreated). A service that does not come back stops the cluster, as one that exits does, and the renewal rolls back. The inventory, web reader and administrator credentials are read on every use and need no restart. A stopped cluster is started instead, for the proof, and stopped again at the end;
each new credential is proved to work for its purpose: a store serves a namespace read (the inventory, read as scribe-monitor or else the gateway's account), which it authorizes through auth with its own new credential; the gateway reports a poll of every instance made after the proof began; the inventory and web reader credentials each read every store's namespace themselves (the web reader through auth's published address, since its file names auth as the containers reach it); the administrator credential answers an administrator request;
only then is each replaced credential revoked and its kept copy removed.

text

$ scribe-ctl --config ./scribe-cluster.toml renew --plan
Scribe · local · renew plan · gateway variant · native runtime
  Definition  /srv/scribe/scribe-cluster.toml
  Paths below are relative to its directory.

  ENDPOINTS
    auth       tls://127.0.0.1:7443
    storage-a  tls://127.0.0.1:7447
    gateway    tls://127.0.0.1:7557

  CHECKS
    ok    credentials   2 of 4 credentials expire within 30 days: the storage-a server credential expires on 2026-09-28; the gateway credential (scribe-gateway) expires on 2026-09-28
    ok    certificates  3 server certificates are valid for 30 more days; the first to expire is auth's on 2027-09-27
    ok    cluster       running; storage-a, gateway restart to read the renewed credentials and certificates, one at a time while the rest serve
    ok    permissions   10 existing secret files and auth's data root are private

  STEPS
     1. Issue a new storage-a server credential for 365 days into secrets/storage-a.credential.next
        it expires on 2026-09-28; the old one stays valid until it is revoked
     2. Issue a new gateway credential (scribe-gateway) for 365 days into secrets/gateway.credential.next
        it expires on 2026-09-28; the old one stays valid until it is revoked
     3. Put the new storage-a server credential in place as secrets/storage-a.credential, keeping the current one as secrets/storage-a.credential.previous
        the holder reads this file; a rollback puts the kept copy back
     4. Put the new gateway credential (scribe-gateway) in place as secrets/gateway.credential, keeping the current one as secrets/gateway.credential.previous
        the holder reads this file; a rollback puts the kept copy back
     5. Restart storage-a, gateway so they read their renewed files
        they read these files only when they start; the supervisor restarts each in turn while the others serve
     6. Verify the new storage-a server credential: storage-a serves a namespace read, which it authorizes with it
        the old credential is revoked only once the new one is proved to work
     7. Verify the new gateway credential (scribe-gateway): the gateway polls every instance with it
        the old credential is revoked only once the new one is proved to work
     8. Revoke the replaced storage-a server credential and remove its kept copy secrets/storage-a.credential.previous
        nothing uses it any more
     9. Revoke the replaced gateway credential (scribe-gateway) and remove its kept copy secrets/gateway.credential.previous
        nothing uses it any more

  ROLLBACK (if one of steps 3-7 fails)
    10. Put the previous gateway credential (scribe-gateway) back from secrets/gateway.credential.previous into secrets/gateway.credential
        the old credential is still valid
    11. Put the previous storage-a server credential back from secrets/storage-a.credential.previous into secrets/storage-a.credential
        the old credential is still valid
    12. Restart storage-a, gateway so they read the previous files again
        back to the files that served before
    13. Revoke the new storage-a server credential this renewal issued
        an unproved credential is not left valid
    14. Revoke the new gateway credential (scribe-gateway) this renewal issued
        an unproved credential is not left valid

  NOTES
    - Nothing old stops working before its replacement is proved; if a verification fails, the old files go back in place and the new credentials are revoked.

  Plan only: nothing was written and no lock was taken. Run renew without --plan to apply it.
$ scribe-ctl --config ./scribe-cluster.toml renew
  local · [1/9] Issue a new storage-a server credential for 365 days into secrets/storage-a.credential.next…
  …
  local · [9/9] Revoke the replaced gateway credential (scribe-gateway) and remove its kept copy secrets/gateway.credential.previous…
Scribe · local · Renewal complete
  storage-a: issued a new credential for 365 days into secrets/storage-a.credential.next
  scribe-gateway: issued a new credential for 365 days into secrets/gateway.credential.next
  put the new credential in place as secrets/storage-a.credential; the current one is kept as secrets/storage-a.credential.previous until it is revoked
  put the new credential in place as secrets/gateway.credential; the current one is kept as secrets/gateway.credential.previous until it is revoked
  restarted storage-a, gateway, which read their renewed files
  storage-a: served a namespace read (0 repositories), authorized with the new credential
  gateway: polled every instance with the new credential
  storage-a: revoked the replaced credential dfbb4e5c3a71
  scribe-gateway: revoked the replaced credential 3eaa5e0b51fa

  RENEWED
    storage-a server credential: secrets/storage-a.credential, valid until 2027-09-27
    gateway credential (scribe-gateway): secrets/gateway.credential, valid until 2027-09-27

If a step from the first put-in-place through the last proof fails, renew rolls back: the kept credentials go back in place, the holders restart with them (or the cluster is stopped again), and the new credentials are revoked. It exits 1; the old credentials stay valid until their own expiry, and the next run tries again:

text

$ scribe-ctl --config ./scribe-cluster.toml renew
  local · [1/4] Issue a new inventory credential (scribe-monitor) for 365 days into secrets/monitor.credential.next…
  local · [2/4] Put the new inventory credential (scribe-monitor) in place as secrets/monitor.credential, keeping the current one as secrets/monitor.credential.previous…
  local · [3/4] Verify the new inventory credential (scribe-monitor): an inventory read with it…
  local · [5/6] Put the previous inventory credential (scribe-monitor) back from secrets/monitor.credential.previous into secrets/monitor.credential…
  local · [6/6] Revoke the new inventory credential (scribe-monitor) this renewal issued…
Scribe · local · Renewal rolled back
  step 3 (Verify the new inventory credential (scribe-monitor): an inventory read with it) failed: …
  scribe-monitor: issued a new credential for 365 days into secrets/monitor.credential.next
  put the new credential in place as secrets/monitor.credential; the current one is kept as secrets/monitor.credential.previous until it is revoked

  ROLLED BACK
    put the previous inventory credential (scribe-monitor) back into secrets/monitor.credential
    scribe-monitor: revoked the new credential aa64db7532cf

  The previous inventory credential (scribe-monitor) is in place again, valid until 2026-09-28.
  Fix the cause, then run renew again.

A failure before anything is put in place (auth unreachable while issuing) leaves the old credentials in place and the issued ones in <file>.next; a failure while revoking leaves the proved new credentials in place and the old ones still valid. Either exits 1, and the next run resumes where it stopped.

Server certificates

A due certificate is reissued for the key it already names (the server's private key is never read or replaced), for the names the definition gives the service, by the CA that signed it, for 365 days (never past the CA itself; a CA with less than 30 days left is refused: rotate it first). The file holds the certificate followed by its CA, so what the server presents ends with the CA clients trust and fingerprint. The file is put in place like a credential (kept as <file>.previous), only that service restarts, and a TLS handshake trusting the CA alone, as a client does, proves it before the kept copy is removed; clients notice nothing:

text

$ scribe-ctl --config ./scribe-cluster.toml renew --plan
Scribe · local · renew plan · gateway variant · native runtime
  Definition  /srv/scribe/scribe-cluster.toml
  Paths below are relative to its directory.

  ENDPOINTS
    auth       tls://127.0.0.1:7443
    storage-a  tls://127.0.0.1:7447
    gateway    tls://127.0.0.1:7557

  CHECKS
    ok    credentials   nothing to renew: 4 credentials are valid for 30 more days; the first to expire is the storage-a server credential on 2027-09-27
    ok    certificates  1 of 3 server certificates expire within 30 days: storage-a's expires on 2026-10-09; each is reissued for the key it has
    ok    cluster       running; storage-a restarts to read the renewed credentials and certificates, one at a time while the rest serve
    ok    permissions   10 existing secret files and auth's data root are private

  STEPS
     1. Reissue storage-a's server certificate for its key into pki/storage-a/server.pem.next, signed with the CA key in pki/ca, followed by its CA
        it expires on 2026-10-09; the same key, the same CA
     2. Put storage-a's new certificate in place as pki/storage-a/server.pem, keeping the current one as pki/storage-a/server.pem.previous
        the server reads this file when it starts; a rollback puts the kept copy back
     3. Restart storage-a so it reads its renewed files
        they read these files only when they start; the supervisor restarts each in turn while the others serve
     4. Verify a TLS handshake with storage-a trusting the current storage and gateway CA alone
        clients hold one CA each; every one of them must still connect
     5. Remove storage-a's replaced certificate pki/storage-a/server.pem.previous
        the new certificate is proved

  ROLLBACK (if one of steps 2-4 fails)
     6. Put storage-a's previous certificate back from pki/storage-a/server.pem.previous into pki/storage-a/server.pem
        the old certificate is still valid
     7. Restart storage-a so it reads the previous files again
        back to the files that served before

  NOTES
    - Nothing old stops working before its replacement is proved; if a verification fails, the old files go back in place and the new credentials are revoked.

  Plan only: nothing was written and no lock was taken. Run renew without --plan to apply it.
$ scribe-ctl --config ./scribe-cluster.toml renew
  local · [1/5] Reissue storage-a's server certificate for its key into pki/storage-a/server.pem.next, signed with the CA key in pki/ca, followed by its CA…
  …
  local · [5/5] Remove storage-a's replaced certificate pki/storage-a/server.pem.previous…
Scribe · local · Renewal complete
  issued a new certificate for pki/storage-a/server.pem's key into pki/storage-a/server.pem.next, valid for 365 days
  put the new certificate in place as pki/storage-a/server.pem; the current one is kept as pki/storage-a/server.pem.previous
  restarted storage-a, which reads its renewed files
  storage-a: TLS handshake at 127.0.0.1:7447 as localhost verified trusting the current storage and gateway CA alone
  storage-a: removed the replaced certificate pki/storage-a/server.pem.previous

  RENEWED
    storage-a's certificate: pki/storage-a/server.pem, valid until 2027-09-27

The CA's key signs where renew runs. Setup keeps it beside the cluster (ca-key.pem in the CA directory); moved elsewhere, name its directory with --ca-dir DIR (the storage and gateway CA) or --auth-ca-dir DIR (auth's). Without it the plan fails for exactly that certificate and names the command:

text

$ scribe-ctl --config ./scribe-cluster.toml renew --plan
  …
  CHECKS
    ok    credentials   nothing to renew: 4 credentials are valid for 30 more days; the first to expire is the storage-a server credential on 2027-09-27
    ok    certificates  1 of 3 server certificates expire within 30 days: storage-a's expires on 2026-10-09; each is reissued for the key it has
    ok    cluster       running; storage-a restarts to read the renewed credentials and certificates, one at a time while the rest serve
    ok    permissions   9 existing secret files and auth's data root are private
    FAIL  ca keys       the storage and gateway CA's key is not beside the cluster (pki/ca/ca-key.pem is missing): run `scribe-ctl --config /srv/scribe/scribe-cluster.toml renew --ca-dir DIR` on the machine that holds it, where DIR holds its ca.pem and ca-key.pem (with --remote from there, only the new certificates travel to the cluster)
  …

With --remote, DIR is on the machine you run scribe-ctl on: the remote scribe-ctl says what needs signing (each certificate's public part and names), this machine signs with the key in DIR and pushes only the certificates (upload/renew/signed.json in the cluster's directory), and the remote checks each (the same key, issued by its CA, exactly its names) before it puts it in place. The key never leaves this machine. --plan with --ca-dir shows the remote plan as if this machine signs.

Server keys

A plain renew keeps each server's private key. --rekey [SERVICE,...] replaces it, for a key suspected leaked or to move a server to a post-quantum (ML-DSA-65) key: for each named TLS server (every one when none is named) a new key is generated beside the current one (<key>.next, owner-only, written atomically), the CA that signs the server now certifies it for the same names (followed by the cross certificate during a rotation's overlap, as a reissue is), key and certificate are put in place together (the old ones kept as <file>.previous), only that service restarts, and a TLS handshake trusting the CA clients hold proves the new pair before the kept key and certificate are removed. A failure from the key's put-in-place through the handshake puts the old key and certificate back together. Clients trust the CA, not the key, so none of them changes anything. A rekey run does only the rekey; nothing that expires is renewed alongside.

text

$ scribe-ctl --config ./scribe-cluster.toml renew --rekey storage-a
  local · [1/8] Generate a new private key for storage-a into pki/storage-a/server-key.pem.next (only this account can read it)…
  local · [2/8] Issue storage-a's server certificate for its new key pki/storage-a/server-key.pem.next into pki/storage-a/server.pem.next, signed with the CA key in pki/ca, followed by its CA…
  local · [3/8] Put storage-a's new private key in place as pki/storage-a/server-key.pem, keeping the current one as pki/storage-a/server-key.pem.previous…
  local · [4/8] Put storage-a's new certificate in place as pki/storage-a/server.pem, keeping the current one as pki/storage-a/server.pem.previous…
  local · [5/8] Restart storage-a so it reads its renewed files…
  local · [6/8] Verify a TLS handshake with storage-a trusting the current storage and gateway CA alone…
  local · [7/8] Remove storage-a's replaced certificate pki/storage-a/server.pem.previous…
  local · [8/8] Remove storage-a's replaced private key pki/storage-a/server-key.pem.previous…
Scribe · local · Renewal complete
  …
  RENEWED
    storage-a's certificate: pki/storage-a/server.pem, valid until 2027-09-28, for a new key

A name that is not a TLS server of the cluster fails the plan (nope is not a TLS server of this cluster; its TLS servers are auth, storage-a). With --remote and --ca-dir DIR, the remote scribe-ctl stages the new key there first and asks for its public key to be certified (public_key in its request); this machine certifies it with the key in DIR, and only the certificate travels back. The new private key never leaves the cluster's machine. A <key>.next a request staged and no run used is what the next --rekey of that service certifies and puts in place; doctor reports it (and any .next or .previous beside a certificate or key with no renewal pending) as renew.leftovers.<service>.

CA rotation

--rotate-ca storage|auth replaces a CA through a signed handover, never started by a plain renew (which only warns, from a year before the CA expires). It creates a new CA and a cross certificate: the new CA's name and key, a CA, issued by the old CA, valid for the overlap (90 days, or --overlap-days, never past the old CA). Every server certificate of that CA is reissued in the same run by the new CA for the key it has, followed by the cross certificate, so a client that trusts only the old CA verifies it through the cross certificate (and learns the new CA), and a client given the new CA verifies it directly. Before any server changes, the web interface's services that read /etc/scribe/trust restart trusting both CAs (Node follows no cross certificate: it refuses a chain through one to an old CA that may sign no intermediate). The servers restart, handshakes trusting the old and the new CA alone prove them, then the new CA becomes ca.pem in the CA directory (the old one and its key move to previous/, the cross certificate to cross.pem), the trust files the definition names beside it (a store's ca, gateway instances' ca, auth's ca, a ca.pem beside a server certificate) hold both CAs, for the auth CA every credential file ctl manages is rewritten to carry the new auth CA, and the services that read the new trust restart (the gateway for the storage CA; the stores and the gateway for auth's; the web services). The rotation is recorded in the cluster's state directory (ca-rotation.json) with the day it retires. Any failure from the first certificate put in place through the last proof puts the old CA and certificates back and sets the new CA aside:

text

$ scribe-ctl --config ./scribe-cluster.toml renew --rotate-ca storage --plan
Scribe · local · renew plan · gateway variant · native runtime
  Definition  /srv/scribe/scribe-cluster.toml
  Paths below are relative to its directory.

  ENDPOINTS
    auth       tls://127.0.0.1:7443
    storage-a  tls://127.0.0.1:7447
    gateway    tls://127.0.0.1:7557

  CHECKS
    ok    rotation     starts a rotation of the storage and gateway CA: 2 server certificates (storage-a, gateway) move to a new CA together, followed by a cross certificate the old CA signs, so clients that trust only the old CA keep connecting for 90 days
    ok    cluster      running; storage-a, gateway restart to read the renewed credentials and certificates, one at a time while the rest serve
    ok    permissions  10 existing secret files and auth's data root are private

  STEPS
     1. Create the new storage and gateway CA in pki/ca/next and the cross certificate the old one signs for it, valid for the 90-day overlap
        the old CA signs the cross certificate that hands its trust to the new one
     2. Reissue storage-a's server certificate for its key into pki/storage-a/server.pem.next, signed with the CA key in pki/ca/next, followed by the cross certificate
        every server of the CA moves to the new one in this rotation
     3. Reissue gateway's server certificate for its key into pki/gateway/server.pem.next, signed with the CA key in pki/ca/next, followed by the cross certificate
        every server of the CA moves to the new one in this rotation
     4. Put storage-a's new certificate in place as pki/storage-a/server.pem, keeping the current one as pki/storage-a/server.pem.previous
        the server reads this file when it starts; a rollback puts the kept copy back
     5. Put gateway's new certificate in place as pki/gateway/server.pem, keeping the current one as pki/gateway/server.pem.previous
        the server reads this file when it starts; a rollback puts the kept copy back
     6. Restart storage-a, gateway so they read their renewed files
        they read these files only when they start; the supervisor restarts each in turn while the others serve
     7. Verify a TLS handshake with storage-a trusting the current storage and gateway CA, then the new storage and gateway CA alone
        clients hold one CA each; every one of them must still connect
     8. Verify a TLS handshake with gateway trusting the current storage and gateway CA, then the new storage and gateway CA alone
        clients hold one CA each; every one of them must still connect
     9. Make the new storage and gateway CA current in pki/ca (the old one moves to previous/) and record the rotation
        every server presents the new CA's certificates with the cross certificate
    10. Restart gateway so it reads the new CA
        they read the CA only when they start
    11. Verify a TLS handshake with storage-a trusting the current storage and gateway CA, then the old storage and gateway CA alone
        clients that learned the new CA and clients that know only the old one both connect
    12. Verify a TLS handshake with gateway trusting the current storage and gateway CA, then the old storage and gateway CA alone
        clients that learned the new CA and clients that know only the old one both connect
    13. Read through every store with a reader credential, and wait for the gateway's fresh inventory
        every store serves reads through the new trust
    14. Remove storage-a's replaced certificate pki/storage-a/server.pem.previous
        the new certificate is proved
    15. Remove gateway's replaced certificate pki/gateway/server.pem.previous
        the new certificate is proved

  ROLLBACK (if one of steps 4-13 fails)
    16. Put gateway's previous certificate back from pki/gateway/server.pem.previous into pki/gateway/server.pem
        the old certificate is still valid
    17. Put storage-a's previous certificate back from pki/storage-a/server.pem.previous into pki/storage-a/server.pem
        the old certificate is still valid
    18. Make the old storage and gateway CA current again in pki/ca
        the old CA served before the rotation
    19. Move the unused new storage and gateway CA aside in pki/ca
        a later rotation stages a CA of its own
    20. Restart storage-a, gateway so they read the previous files again
        back to the files that served before

  NOTES
    - Nothing old stops working before its replacement is proved; if a verification fails, the old files go back in place and the new credentials are revoked.
    - After the rotation, clients learn the new storage and gateway CA whenever they connect during the 90-day overlap; `scribe-ctl --config /srv/scribe/scribe-cluster.toml status` shows the day a plain renew retires the old one. Clients that never connect in that time see "CA changed" and trust the new CA by hand.

  Plan only: nothing was written and no lock was taken. Run renew without --plan to apply it.
$ scribe-ctl --config ./scribe-cluster.toml renew --rotate-ca storage
  local · [1/15] Create the new storage and gateway CA in pki/ca/next and the cross certificate the old one signs for it, valid for the 90-day overlap…
  …
  local · [15/15] Remove gateway's replaced certificate pki/gateway/server.pem.previous…
Scribe · local · Renewal complete
  created the new storage and gateway CA and its cross certificate in pki/ca/next
  issued a new certificate for pki/storage-a/server.pem's key into pki/storage-a/server.pem.next, valid for 365 days
  issued a new certificate for pki/gateway/server.pem's key into pki/gateway/server.pem.next, valid for 365 days
  put the new certificate in place as pki/storage-a/server.pem; the current one is kept as pki/storage-a/server.pem.previous
  put the new certificate in place as pki/gateway/server.pem; the current one is kept as pki/gateway/server.pem.previous
  restarted storage-a, gateway, which read their renewed files
  storage-a: TLS handshake at 127.0.0.1:7447 as localhost verified trusting the current storage and gateway CA, and the new storage and gateway CA alone
  gateway: TLS handshake at 127.0.0.1:7557 as localhost verified trusting the current storage and gateway CA, and the new storage and gateway CA alone
  the new storage and gateway CA is current; the old one stays trusted through the cross certificate until 2026-12-26, when a plain renew retires it
  restarted gateway, which reads the new CA
  storage-a: TLS handshake at 127.0.0.1:7447 as localhost verified trusting the current storage and gateway CA, and the old storage and gateway CA alone
  gateway: TLS handshake at 127.0.0.1:7557 as localhost verified trusting the current storage and gateway CA, and the old storage and gateway CA alone
  storage-a: serves its repositories (0) from /srv/scribe/data/storage-a
  gateway: polled every instance
  storage-a: removed the replaced certificate pki/storage-a/server.pem.previous
  gateway: removed the replaced certificate pki/gateway/server.pem.previous

  RENEWED
    storage-a's certificate: pki/storage-a/server.pem, valid until 2027-09-27, from the new CA
    gateway's certificate: pki/gateway/server.pem, valid until 2027-09-27, from the new CA

  ROTATION
    The new storage and gateway CA is current: BLAKE3:a82af9cb26c76557c03c762ac30ff2c91473706a8f808574a7ed93859c2c67bc
    The old one (BLAKE3:8ca70e3c41276a59c247660fd4aa122117645038abc2e25ce8ba18560dd80f8c) stays trusted through the cross certificate for 90 days; a plain renew then retires it.
    Clients learn the new CA when they connect in that time; give new clients the new ca.pem.

status shows the rotation (and doctor its ca.rotation.storage check); a plain renew in the overlap changes nothing and says when it retires:

text

$ scribe-ctl --config ./scribe-cluster.toml status
Scribe · local · running
  SERVICE    STATE        PID  ENDPOINT
  auth       running    46876  tls://127.0.0.1:7443
  storage-a  running    46907  tls://127.0.0.1:7447
  gateway    running    46920  tls://127.0.0.1:7557

  Logs  /srv/scribe/cluster-state/local/logs
  Binaries  0.1.0 · /srv/scribe/cluster-state/local/bin/0.1.0
  CA  the storage and gateway CA is being rotated since 2026-09-27; a plain renew retires the old CA on 2026-12-26 (`scribe-ctl --config /srv/scribe/scribe-cluster.toml renew --retire-ca storage` does it now)
$ scribe-ctl --config ./scribe-cluster.toml renew
Scribe · local · Unchanged: nothing to renew: 4 credentials are valid for 30 more days; the first to expire is the storage-a server credential on 2027-09-27
  3 server certificates are valid for 30 more days; the first to expire is auth's on 2027-09-27
  The storage and gateway CA is being rotated: its old CA stays trusted until 2026-12-26, when a plain renew retires it (`scribe-ctl --config /srv/scribe/scribe-cluster.toml renew --retire-ca storage` does it now).

From the overlap's end, the scheduled plain renew retires the old CA: each certificate is followed by the new CA itself instead of the cross certificate, previous/ and cross.pem move to retired-<date>-<old fingerprint>/ (the key is kept there, never deleted: move it offline or remove it), the trust files hold the new CA alone, the servers and the web services restart, and handshakes trusting the new CA prove them. A client that never connected during the overlap sees "CA changed" with both fingerprints and trusts the new CA by hand (CA rotation and handover):

text

$ scribe-ctl --config ./scribe-cluster.toml renew --plan
Scribe · local · renew plan · gateway variant · native runtime
  Definition  /srv/scribe/scribe-cluster.toml
  Paths below are relative to its directory.

  ENDPOINTS
    auth       tls://127.0.0.1:7443
    storage-a  tls://127.0.0.1:7447
    gateway    tls://127.0.0.1:7557

  CHECKS
    ok    credentials   nothing to renew: 4 credentials are valid for 30 more days; the first to expire is the storage-a server credential on 2027-09-27
    ok    rotations     the storage and gateway CA's overlap ended on 2026-09-27: its old CA is retired now (clients that have not connected since 2026-09-27 are asked to trust the new one)
    ok    certificates  3 server certificates are valid for 30 more days; the first to expire is auth's on 2027-09-27
    ok    cluster       running; storage-a, gateway restart to read the renewed credentials and certificates, one at a time while the rest serve
    ok    permissions   10 existing secret files and auth's data root are private

  STEPS
     1. Write storage-a's certificate followed by the new CA instead of the cross certificate into pki/storage-a/server.pem.next
        after the overlap, clients that trust only the old CA are no longer served
     2. Write gateway's certificate followed by the new CA instead of the cross certificate into pki/gateway/server.pem.next
        after the overlap, clients that trust only the old CA are no longer served
     3. Put storage-a's new certificate in place as pki/storage-a/server.pem, keeping the current one as pki/storage-a/server.pem.previous
        the server reads this file when it starts; a rollback puts the kept copy back
     4. Put gateway's new certificate in place as pki/gateway/server.pem, keeping the current one as pki/gateway/server.pem.previous
        the server reads this file when it starts; a rollback puts the kept copy back
     5. Retire the old storage and gateway CA: archive previous/ and the cross certificate as retired-2026-09-27-8ca70e3c/ in pki/ca (its key is kept, never deleted)
        every server presents the new CA itself from now on
     6. Restart storage-a, gateway so they read their renewed files
        they read these files only when they start; the supervisor restarts each in turn while the others serve
     7. Verify a TLS handshake with storage-a trusting the current storage and gateway CA alone
        clients hold one CA each; every one of them must still connect
     8. Verify a TLS handshake with gateway trusting the current storage and gateway CA alone
        clients hold one CA each; every one of them must still connect
     9. Read through every store with a reader credential, and wait for the gateway's fresh inventory
        every store serves reads through the new trust
    10. Remove storage-a's replaced certificate pki/storage-a/server.pem.previous
        the new certificate is proved
    11. Remove gateway's replaced certificate pki/gateway/server.pem.previous
        the new certificate is proved

  ROLLBACK (if one of steps 3-9 fails)
    12. Put gateway's previous certificate back from pki/gateway/server.pem.previous into pki/gateway/server.pem
        the old certificate is still valid
    13. Put storage-a's previous certificate back from pki/storage-a/server.pem.previous into pki/storage-a/server.pem
        the old certificate is still valid
    14. Put the old storage and gateway CA back from retired-2026-09-27-8ca70e3c/ in pki/ca
        the rotation's overlap goes on
    15. Restart storage-a, gateway so they read the previous files again
        back to the files that served before

  NOTES
    - Nothing old stops working before its replacement is proved; if a verification fails, the old files go back in place and the new credentials are revoked.

  Plan only: nothing was written and no lock was taken. Run renew without --plan to apply it.
$ scribe-ctl --config ./scribe-cluster.toml renew
  local · [1/11] Write storage-a's certificate followed by the new CA instead of the cross certificate into pki/storage-a/server.pem.next…
  …
  local · [11/11] Remove gateway's replaced certificate pki/gateway/server.pem.previous…
Scribe · local · Renewal complete
  wrote pki/storage-a/server.pem's certificate followed by pki/ca/ca.pem into pki/storage-a/server.pem.next
  wrote pki/gateway/server.pem's certificate followed by pki/ca/ca.pem into pki/gateway/server.pem.next
  put the new certificate in place as pki/storage-a/server.pem; the current one is kept as pki/storage-a/server.pem.previous
  put the new certificate in place as pki/gateway/server.pem; the current one is kept as pki/gateway/server.pem.previous
  retired the old storage and gateway CA: archived as retired-2026-09-27-8ca70e3c/ in pki/ca (keep its key offline or remove it when you no longer need it)
  restarted storage-a, gateway, which read their renewed files
  storage-a: TLS handshake at 127.0.0.1:7447 as localhost verified trusting the current storage and gateway CA alone
  gateway: TLS handshake at 127.0.0.1:7557 as localhost verified trusting the current storage and gateway CA alone
  storage-a: serves its repositories (0) from /srv/scribe/data/storage-a
  gateway: polled every instance
  storage-a: removed the replaced certificate pki/storage-a/server.pem.previous
  gateway: removed the replaced certificate pki/gateway/server.pem.previous

  RENEWED
    storage-a's certificate: pki/storage-a/server.pem, valid until 2027-09-27, followed by the new CA
    gateway's certificate: pki/gateway/server.pem, valid until 2027-09-27, followed by the new CA

  RETIRED
    The old storage and gateway CA (BLAKE3:8ca70e3c41276a59c247660fd4aa122117645038abc2e25ce8ba18560dd80f8c) is trusted nowhere any more; archived as retired-2026-09-27-8ca70e3c/ (its key is kept: move it offline or remove it).

--retire-ca storage|auth retires a rotation before its overlap ends, with that warning in its plan. With the key outside the cluster, --ca-dir DIR creates the new CA in DIR/next/, and after the run DIR follows the cluster (the new CA current, the old one in DIR/previous/, archived when the rotation retires with --ca-dir DIR); with --remote the new CA's public part and the certificates travel, never a key.

--rotate-ca credential|access adds a successor to one of auth's internal client CAs, which only Scribe's services trust: the cluster drains, scribe-auth authorities add runs on the stopped auth root, and the cluster starts again and serves reads; the new CA signs 15 minutes after it was added. --retire-ca credential|access retires the oldest the same way once nothing it signed is in use (auth refuses before and says why). See rotating the internal CAs.

Runs, resumes and schedulers

Renew works while the cluster runs (the normal case) and under both runtimes. It takes the command lock and journals every step like setup and upgrade: an interrupted renewal resumes when run again, as a plain renew whatever options started it, adopting a new credential auth issued before the interruption from its <file>.next (never issuing a second), a staged CA, and a rotation's switch half done; a rollback that stopped is finished from the journal alone. Other mutating commands refuse to run while a renewal is pending, and renew refuses while another one is. Crash windows inside it are tested end to end (ctl-renewal-issued:<file>, ctl-credential-kept:<file>, ctl-credential-restored:<file>, ctl-replaced-revoked:<file>, ctl-certificate-kept:<file>, ctl-key-kept:<file>, ctl-key-restored:<file>, ctl-next-ca-key-written:<CA>, ctl-ca-set-aside:<CA>, ctl-ca-archived:<CA> besides the step windows; test builds only).

For schedulers: renew never prompts, writes one line per step on stderr and its result on stdout, and exits 0 (renewed, or nothing due), 1 (a check or step failed, or the renewal was rolled back: the files in place work in every case) or 2 (invalid arguments). With --json the result is one object: renewed (per credential holder, credential, file, replaced, replaced_expires_ms, new, expires_ms), certificates (per file service, file, key (the private key a rekey replaced, else null), change of reissue, handover, rechain or rekey, replaced_expires_ms, expires_ms), rotation, retired and authority, or unchanged: true, or on a rollback rolled_back, failure, rollback and kept. status --json lists ca_rotations. Renew installs no schedule; the cluster guide has cron, systemd timer and Task Scheduler examples. Run it daily: it is a no-op until 30 days before an expiry or the end of an overlap.

adopt

Take over a Docker deployment scribe-ctl did not create (docker/init.sh's, or Compose projects run by hand) without moving its data or secrets: find its services, data, secrets, certificates and ports from its containers, write a definition that points at them where they are (the web interface's secrets and certificate are copied into the cluster's directory), then stop the old Compose projects (never removing them or their volumes) and start the cluster's own; a failed health check stops it and starts the old projects again as they were. Resumable

text

scribe-ctl adopt [OPTIONS] --runtime <RUNTIME>

Argument or option	Meaning
`--runtime <RUNTIME>`	Where the deployment runs: docker (Compose projects on this machine, or with --remote on that one). A native deployment is adopted with a hand-written definition and setup instead (docs/cluster-usage.md)
`--project <NAME>`	A Compose project to take over: the one that runs Scribe's services, and again for the web interface's when it runs in another. Without it, adopt takes the one project running Scribe services that scribe-ctl does not manage, with every project on its networks, and refuses when that is ambiguous
`--admin-credential <FILE>`	Auth's bootstrap administrator credential, which no service container mounts (docker/init.sh wrote operator/admin.credential in its state directory). Required when the deployment runs auth; it stays where it is
`--source <DIR>`	The Scribe checkout (of this scribe-ctl's version) to build the images from, as setup does. Needed unless the running images are this scribe-ctl's release, and always with the web interface (the image set keeps its Compose file). Also builds with --plan; nothing in the deployment changes
`--plan`	Show the plan and run every read-only check (containers, files, credentials, formats, ports, auth), without writing anything, taking a lock or touching a container

adopt is for a deployment that already serves: the Compose projects the removed docker/init.sh made (basic, auth or gateway mode), a gateway deployment with the web interface's own Compose project beside it, or the same shapes started by hand. It reads everything from docker inspect of the projects' containers, never from their names alone:

Which projects. Named with --project (the Scribe services' project, and again the web interface's), or found: the one Compose project that runs scribe-auth, scribe-server or scribe-gateway serve and that scribe-ctl does not run itself, plus every project with a container on one of its networks (the web interface's joins it). Two candidates are refused with both named.
What each container is. Auth, a store and the gateway by the process they run; the web interface's services by their Compose service being one of apps/web's (which is why the web interface needs --source). Any other container, a flag scribe-ctl's containers would not pass (--session-ttl-days, --log-dir, …), a storage configuration setting other than the data root, listener, auth namespace, credential and certificate, a SCRIBE_* variable, a second container of one service, or a data root in a named volume is refused by name: scribe-ctl never drops a setting silently.
Where everything is. Every container path maps to its host path through the container's bind mounts: data roots, pepper and witness, certificates and keys, each credential, the gateway's configuration and trust files, the web interface's secrets, TLS directory and reader credential. Published ports become listen addresses; a store the deployment did not publish (docker/init.sh's gateway mode) is published on loopback at its container port, so inventory and status can read it. Each certificate's CA comes from the ca.pem beside it (for auth, else from the credentials, which carry it); a sibling directory with that ca.pem and its ca-key.pem becomes the definition's CA directory, otherwise a copy of the certificate alone is written beside the definition and generate_missing is off.

The definition it writes names those files where they are (absolute paths, outside the definition's directory), with runtime = "docker" and the variant the services form, except the web interface's secrets and certificate. Those often live in a checkout's apps/web/deploy/local, which may be replaced or deleted, so adopt copies them where setup keeps its own: the session key and database password to secrets/web/, the reader credential alone to secrets/web-reader/scribe-web.credential, and fullchain.pem and key.pem (when the site serves HTTPS) to secrets/web-tls/. The copy is a step before the old projects stop, so it adds no downtime: each file is read whole, written privately (0600, its directory 0700, owned by the definition directory's owner, as whom the containers run), synced, renamed into place and read back. The originals are never moved or changed, so the old project and a rollback keep working; a copy already there and identical is kept, and a different file where a copy goes is refused. Beside the definition scribe-ctl also writes only what it owns: cluster-state/NAME/, the gateway's configuration with host paths and endpoints (scribe-gateway.toml), the CA copies when the keys were not found, secrets/monitor.credential (the scribe-monitor inventory account, as setup provisions it) and, when the web database lived in a named volume, data/web. Three definition keys exist for adopted deployments: [auth] container_port (the port auth's credentials name as tls://auth:PORT, when it is not the published one), [web] clone_host (the host name the Clone menu gave clients) and [web] volumes (the web interface's named volumes kept as they are, such as objects).

The checks, besides discovery's: the images serve the data as it is (every data root's formats equal to what the images read and write; auth schema 3 is refused because the identity v2 switch has not happened), the revision witness and pepper exist, every secret is private and so is auth's data root (auth refuses to start from a root other accounts can enter, so a mode like 0755 fails the plan with the chmod 700 <host path> that fixes it, before anything is stopped), every certificate loads, every credential names auth as the cluster's containers reach it and trusts the same auth CA (an expired one is refused, one expiring within 30 days noted for renew), the services ran as the user scribe-ctl's containers run as (the owner of the definition's directory), ports scribe-ctl newly publishes are free, auth answers the administrator credential, and the web interface's variables would not change (its container environments compared with what docker compose config renders for scribe-ctl's project; WEB_TLS must match whether the TLS directory holds fullchain.pem and key.pem), and the web secrets to copy can be read, with no different file already where a copy goes.

Then, for a gateway deployment with the web interface in its own project (docker/init.sh gateway with apps/web's Compose files beside it; paths shortened):

text

$ scribe-ctl --config /srv/scribe-cluster/scribe-cluster.toml --cluster prod adopt --runtime docker \
    --admin-credential /srv/old-scribe/operator/admin.credential --source /srv/src --plan
  prod · Building scribe/auth:0.1.0 from /srv/src (the first build takes a while)…
  …
Scribe · prod · adopt plan · gateway variant · docker runtime
  …
  CHECKS
    ok    deployment   Compose project scribe-gateway (running auth, gateway, storage-a; from /srv/scribe/docker/gateway/compose.yaml); Compose project scribe-web (running api, assets-derive, assets-derive-sandbox-image, assets-derive-sandbox-scene, assets-index, cache, content, database, nginx, objects, search, web, worker; from /srv/scribe/apps/web/compose.yaml,/srv/scribe/apps/web/compose.scribe.yaml)
    ok    definition   adopt writes cluster prod into scribe-cluster.toml
    ok    binaries     scribe-auth, scribe-server, scribe-gateway 0.1.0; setup records them as the first image set in cluster-state/prod/bin/0.1.0
    ok    permissions  12 existing secret files and auth's data root are private
    ok    data         in place, served as they are: auth auth_schema 4; storage-a catalog_schema 1, envelope 1, object_store 1, repository_schema 3; gateway route_store_schema 2
    ok    credentials  4 credentials name auth as tls://auth:7443 and trust its CA
    ok    user         the services run as 501:20, the owner of the definition's directory, as scribe-ctl's containers do
    ok    ports        the old project's published ports are free once it stops; also published now: 127.0.0.1:7447 (free)
    ok    auth         answers the administrator at tls://127.0.0.1:27443
    ok    web          project scribe-web at http://localhost:28080; its database is copied once from the volume scribe-web_database into data/web (the volume is kept)
    ok    web secrets  5 files to copy into secrets/web, secrets/web-reader, secrets/web-tls; the originals stay where they are
  …
  STEPS
     1. Write the cluster definition scribe-cluster.toml and scribe-gateway.toml
     2. Record the images scribe/auth:0.1.0, scribe/storage:0.1.0, scribe/gateway:0.1.0 and the web interface's of 0.1.0 as a set in cluster-state/prod/bin/0.1.0
     3. Record 0.1.0 as the active set (bin/current)
     4. Copy the web interface's secrets into secrets/web, secrets/web-reader, secrets/web-tls (5 files; the originals stay where they are)
     5. Stop the old Compose projects scribe-web, then scribe-gateway (containers, networks and volumes are kept)
     6. Copy the web database from the volume scribe-web_database into data/web (the volume stays as it is)
     7. Start the cluster's own Compose project with 0.1.0 and wait until every service is ready
     8. Ensure service account scribe-monitor reads in studio
     9. Ensure scribe-monitor's credential secrets/monitor.credential
    10. Read through every store with a reader credential, and wait for the gateway's fresh inventory
    11. Write the cluster summary CLUSTER-INFO.txt

  ROLLBACK (if one of steps 5-10 fails)
    12. Drain whatever of the cluster runs
    13. Move data/web aside to data/web.adopt-rolled-back
    14. Start the old deployment again: scribe-gateway (auth, gateway, storage-a), then scribe-web (api, assets-derive, …, worker)
    15. Take cluster prod out of scribe-cluster.toml again
  …

(The step reasons are left out above.) The plan also shows the definition and the gateway file it writes as a diff. The images come from --source as with setup; without it, the running images are reused (named scribe/SERVICE:BUILD by a docker tag step) only when each answers --build-info as this scribe-ctl's release, which images older than scribe-ctl never do. The old projects are stopped with docker compose -p NAME stop (the web interface's first), which needs no Compose file, and are never removed: their containers, networks, images and volumes stay. The web database is copied once, stopped, by a throwaway container, compared with diff -r and made the cluster user's; its volume is left as it was. The health check is setup's start (up --wait of every container, then the gateway's fresh inventory) plus a namespace read through every store with the new inventory credential.

A failure from the stop on rolls back: scribe-ctl's project stops, the database copy is moved aside (never deleted), the copies of the web secrets stay (nothing but the adopted cluster reads them; a rerun keeps them), docker compose -p NAME start starts exactly the services that ran before (Scribe's project first, in Compose's dependency order), and the cluster is taken out of the definition again (the file is moved aside when it defined only this cluster, else rewritten with the adopted one kept as .bak), so a rerun starts afresh. Verified with an injected failure after the start:

text

Scribe · prod · Adoption rolled back
  step 10 (Read through every store with a reader credential, and wait for the gateway's fresh inventory) failed: injected failure (ctl-step-fails:verify-serving)
  …
  ROLLED BACK
    drained the cluster
    moved data/web aside to data/web.adopt-rolled-back
    started scribe-gateway, scribe-web again
    moved the definition aside to scribe-cluster.toml.adopt-rolled-back-29d5f12ebe9ce759fb45e8f26090b6b9

  The old deployment (scribe-gateway, scribe-web) runs again as it was; nothing of its data was changed.
  Fix the cause, then run adopt again.

A failure before the stop changes nothing of the deployment; the journal (cluster-state/NAME/journal.json) makes a rerun resume, and a rollback that stopped is finished by the next run. Once adopted, the cluster is an ordinary docker cluster: status, where, access, df, renew and upgrade --source work on it, and clients keep their endpoints, sessions and trust. The completed run lists what is left of the old deployment, for you to remove once the cluster has proven itself:

text

  AFTERWARDS
    - Clients keep their endpoints, sessions and trust: the same ports serve the same data. …
    - The old Compose project scribe-gateway is stopped, not removed: its containers (scribe-gateway-gateway-1, scribe-gateway-storage-a-1, scribe-gateway-auth-1) and networks stay, and so do its files (/srv/scribe/docker/gateway/compose.yaml). Once the cluster has proven itself, remove them with `docker compose -p scribe-gateway down` (never with -v while you might still need its volumes).
    - …
    - The web database was copied out of the volume scribe-web_database, which is kept untouched; remove it with `docker volume rm scribe-web_database` once you no longer need it.
    - The web TLS certificate now lives in /srv/scribe-cluster/secrets/web-tls: whoever renews it must write fullchain.pem and key.pem there from now on (e.g. apps/web/deploy/issue-tls.sh --tls-dir /srv/scribe-cluster/secrets/web-tls); /srv/scribe/apps/web/deploy/local/tls is no longer read.

Renewing the web certificate is yours, as after setup: point whatever renews it (apps/web/deploy/issue-tls.sh --tls-dir, a cron job) at the new tls_dir.

With --remote, adopt runs on that machine like every command (with --source, the checkout is pushed and the scribe-ctl that runs adopt is built there from it in Docker, named by the checkout's digest; see remote operation), and the adopted definition comes back into the local file with its [remote] table, as after setup; --admin-credential names a file on the remote machine.

access

Show who can do what in auth: every account with its kind and state, and each privilege it holds. Read-only, through the administrator credential; auth must be running

text

scribe-ctl access [OPTIONS]

Argument or option	Meaning
`--user <NAME>`	Only this account (username or service account)

The report is auth's own, the one scribe-auth access prints (scopes and access kinds are explained there), read with the cluster's [auth] admin_credential. It changes nothing. A cluster without auth, or with auth stopped, is refused with what to do.

text

$ scribe-ctl --config ./scribe-cluster.toml access
Scribe · local · access
  ACCOUNT         KIND     STATE    SCOPE                  ACCESS          DETAIL
  admin           person   enabled  auth                   administrator   accounts, grants and credentials
  admin           person   enabled  credential bootstrap   user            never expires
  scribe-gateway  service  enabled  namespace studio       reader          every repository, including later ones
  scribe-gateway  service  enabled  namespace studio       delegate        presents users' access certificates (a gateway)
  scribe-gateway  service  enabled  credential scribe-ctl  service-reader  expires 2027-09-27
  scribe-monitor  service  enabled  namespace studio       reader          every repository, including later ones
  scribe-monitor  service  enabled  credential scribe-ctl  service-reader  expires 2027-09-27
$ scribe-ctl --config ./scribe-cluster.toml access --user scribe-monitor --json
{
  "cluster": "local",
  "command": "access",
  "ok": true,
  "rows": [
    {
      "access": "reader",
      "account": "scribe-monitor",
      "detail": "every repository, including later ones",
      "kind": "service",
      "scope": "namespace studio",
      "state": "enabled"
    },
    {
      "access": "service-reader",
      "account": "scribe-monitor",
      "detail": "expires 2027-09-27",
      "kind": "service",
      "scope": "credential scribe-ctl",
      "state": "enabled"
    }
  ]
}

where

List every file the cluster uses and what it is for, its endpoints, and the next commands

text

scribe-ctl where [OPTIONS]

No command-specific options; global and parent-group options still apply.

where is read-only. It groups every file by what to do with it: trust certificates to give clients, CA private keys to back up offline, secrets that never leave the machine, the auth revision witness, configuration, data roots to back up, and runtime state. Secrets appear only as paths. It names each endpoint, where the web interface's own TLS certificate goes, the exact next commands (renewing credentials among them), and credentials or certificates within 30 days of expiry, each with what to do: renew for a service credential. Setup writes the same text to CLUSTER-INFO.txt beside the definition. Without --cluster, it covers every cluster in the file. Continuing the gateway example after setup:

text

$ scribe-ctl --config ./scribe-cluster.toml where
Scribe cluster local · gateway variant
Definition: /srv/scribe/scribe-cluster.toml
Files below are relative to its directory.

Endpoints
  auth       tls://127.0.0.1:7443
  storage-a  tls://127.0.0.1:7447
  gateway    tls://127.0.0.1:7557  (clients connect here)

Trust: public: give these to every client and to the web interface
  pki/ca/ca.pem
      storage and gateway CA certificate; clients verify servers with it (--tls-ca)
  pki/auth-ca/ca.pem
      auth CA certificate; clients verify auth with it at login (--auth-ca)

CA private keys: back these up offline and never share them; setup needs them only to issue server certificates
  pki/ca/ca-key.pem
      signs storage and gateway certificates
  pki/auth-ca/ca-key.pem
      signs auth's certificate

Secrets: private to this machine: never share them or put them in version control
  secrets/auth.pepper
      auth's password pepper; back it up apart from the auth data, which cannot check passwords without it
  secrets/admin.credential
      the bootstrap administrator's credential (scribe-auth --admin-credential)
  pki/auth/server-key.pem
      auth's TLS private key
  secrets/storage-a.credential
      storage storage-a's own credential
  secrets/monitor.credential
      the scribe-monitor inventory credential (df, repo list)
  pki/storage-a/server-key.pem
      storage storage-a's TLS private key
  secrets/gateway.credential
      the scribe-gateway service account's credential
  pki/gateway/server-key.pem
      the gateway's TLS private key

Auth revision witness: keep it outside every auth backup; auth refuses to start behind it
  secrets/auth.revision
      auth's latest security revision

Configuration: no secrets inside; keep it with your deployment
  scribe-cluster.toml
      the cluster definition
  scribe-gateway.toml
      the gateway's configuration
  CLUSTER-INFO.txt
      this summary, rewritten by setup, add and upgrade

Data: back these up (docs/migration-backup-restore.md); never run two copies at once
  data/auth
      auth: accounts, credentials, grants
  data/storage-a
      storage storage-a: repositories
  data/gateway
      gateway: placement ledger

Binaries: upgrade installs releases side by side and switches bin/current; it never deletes a set or a backup
  cluster-state/local/bin/0.1.0
      active binary set: 0.1.0, selected by bin/current; services start from it

Runtime: controller state and logs; keep it while the cluster exists
  cluster-state/local
      supervisor state, installed binary sets, upgrade backups, and the journal of an interrupted setup, add or upgrade
  cluster-state/local/logs
      service logs (`logs SERVICE`)

Web interface
  The web interface's own TLS certificate is yours to provide: fullchain.pem and key.pem in its WEB_TLS_DIR (see docs/web-interface.md#connection-configuration). Its services trust /srv/scribe/pki/ca/ca.pem (WEB_SCRIBE_TLS_CA) and /srv/scribe/pki/auth-ca/ca.pem (WEB_SCRIBE_AUTH_CA).

Next
  1. Start the cluster (and stop it with `down`)
       scribe-ctl --config /srv/scribe/scribe-cluster.toml up
  2. Add a person (the password comes from a private file)
       scribe-auth --admin-credential /srv/scribe/secrets/admin.credential users add alice
       scribe-auth --admin-credential /srv/scribe/secrets/admin.credential users password alice --password-file PASSWORD_FILE
       scribe-auth --admin-credential /srv/scribe/secrets/admin.credential namespace-admins set alice studio
  3. Log in and use it
       scribe --server tls://127.0.0.1:7557 --tls-ca /srv/scribe/pki/ca/ca.pem login alice --auth tls://127.0.0.1:7443 --auth-ca /srv/scribe/pki/auth-ca/ca.pem
       scribe --server tls://127.0.0.1:7557 --tls-ca /srv/scribe/pki/ca/ca.pem repo create game
  4. Check on it
       scribe-ctl --config /srv/scribe/scribe-cluster.toml status
       scribe-ctl --config /srv/scribe/scribe-cluster.toml df

example

Print a complete example cluster definition to stdout

text

scribe-ctl example [OPTIONS]

No command-specific options; global and parent-group options still apply.

up

Start the cluster: auth, then storage, then the optional gateway (and under docker the web interface)

text

scribe-ctl up [OPTIONS]

Argument or option	Meaning
`--daemon`	Detach the supervisor and return after every service is ready
`--foreground`	Keep the supervisor attached, overriding daemon = true in the definition

--daemon and --foreground are mutually exclusive and override the definition's daemon (default false; setup writes true). Before anything starts, up refuses an auth data root other accounts can enter, with the chmod 700 <root> that fixes it (auth itself would refuse it after the stores started). Daemon mode returns after readiness; foreground mode stays attached until shutdown. With --json, foreground mode emits its final result after shutdown. Under docker, up rewrites the project's files and runs docker compose up -d --wait; containers are kept by Docker (restart: unless-stopped), and foreground mode stops the project on Ctrl-C.

down

Cooperatively drain all services, including children of a lost supervisor

text

scribe-ctl down [OPTIONS]

No command-specific options; global and parent-group options still apply. Under docker, down is docker compose stop for the project: every container gets SIGTERM and its two-minute grace period, and is kept (with its log).

status

Show supervisor and service state and expiry warnings; never probes arbitrary saved PIDs

text

scribe-ctl status [OPTIONS]

Argument or option	Meaning
`--storage`	Query live disk usage and repositories on every configured storage server

Without --cluster, status --storage inspects every named cluster. Plain status retains its local lifecycle snapshot without network queries. It also warns about every credential and server certificate that expires within 30 days (or did), and every CA that expires within a year, read from the files themselves, with what to do about it (scribe-ctl renew, or for a CA renew --rotate-ca; see renew), shows each CA rotation in its overlap with the day a plain renew retires it (CA line; ca_rotations with --json), and names the active binary set (with --json, binaries: version, build and dir, or null before one is recorded). When this scribe-ctl is another release than the set's, it says where the cluster's own is. Under docker the table lists the project's containers from docker compose ps (no PIDs), the web interface's too. After an upgrade:

text

$ scribe-ctl --config ./scribe-cluster.toml status
Scribe · local · running
  SERVICE    STATE        PID  ENDPOINT
  auth       running     8980  tls://127.0.0.1:7443
  storage-a  running     8981  tls://127.0.0.1:7447
  gateway    running     8982  tls://127.0.0.1:7557

  Logs  /srv/scribe/cluster-state/local/logs
  Binaries  0.1.0 (build next) · /srv/scribe/cluster-state/local/bin/next
  This scribe-ctl is 0.1.0; the cluster's own is /srv/scribe/cluster-state/local/bin/next/scribe-ctl.

doctor

Check the cluster where it runs, and this machine, without changing anything: controller, tools, ssh (remote), binaries, Docker, disk space, data formats, permissions, certificate and credential expiry, services and ports, interrupted commands, backups and adoption leftovers. Each check is ok, info, warn or fail with a fix. Exit 0 when nothing warns or fails, 1 with warnings only, 3 with any failure (2 stays a usage error). Writes, locks, starts, pushes and installs nothing

text

scribe-ctl doctor [OPTIONS]

No command-specific options; global and parent-group options still apply.

doctor is the health check to run first when something seems wrong, and from a monitor on a schedule. It observes the cluster as every plan does (the same discovery, the runtime's status table, the files' own contents) and judges each thing it sees by the rule the mutating commands plan with, so its verdicts and theirs cannot drift: the owner-only rule auth applies to its data root, the upgrade's free-space reserve, the format comparison upgrade makes, the 30-day expiry window status and renew use. It never writes a file (not even a lock file), never starts, stops or restarts a service, and never repairs anything; each problem comes with the command or change that fixes it. Repository integrity is scribe-server doctor's job, offline (routine health checks).

Checks are grouped in sections, in this order; each has a stable id that a monitor can key on (<…> is a service, store or project name):

Section	Checks
Controller	`ctl` (this build), `definition` (it parses and selects the cluster), `definition.files` (every file it names exists; for a cluster that runs here), and for a remote cluster `tool.ssh`, `tool.tar`, `tool.rsync` (rsync 3, or openrsync and 2.x, which push as tar over ssh)
SSH and the remote	Remote clusters only: `ssh.reach` (with the time to connect and log in), `ssh.login` (key login in batch mode, never a prompt), `ssh.shared` (later commands share the connection), `remote.host`, `remote.directory`, `remote.definition` (the remote copy against this machine's, which the next command pushes), `remote.ctl` (the remote scribe-ctl against this one), `remote.doctor`
Binaries	`binaries.active` (`bin/current` and its set), `binaries.controller` (this scribe-ctl against the cluster's release), `binaries.installed`
Docker	Docker clusters only: `docker.engine`, `docker.compose`, `docker.images` (every image of the active set), `docker.build-cache` (info, a warning above 20 GiB)
Disk space	`disk.backups` (room for an upgrade's backups of every root, by the upgrade's own rule: the copy and max(10 %, 1 GiB) beside it) and `disk.<root>` for auth, each store and the web database (a warning below that reserve)
Data	`data.<root>`: exists, initialized, bound to its namespace, and records the formats the active set reads and writes (newer data or a pending migration fails)
Secrets and permissions	`permissions.auth-root` (auth's own owner-only rule; the fix is the `chmod 700`), `permissions.secrets`, `auth.witness` and `auth.pepper` (present, and outside auth's data root by auth's placement rule)
Certificates and credentials	`expiry.<file>` for every credential, certificate and CA (a warning within 30 days, for a CA within a year, a failure once expired, with what replaces it); `ca.rotation.<storage\|auth>` for a CA rotation (info in its overlap with the day it retires, a warning once overdue, for a staged `next/` no run uses, or a leftover `previous/`); `ca.key.<storage\|auth>` for each server CA's key (a warning while it is a classical key such as ECDSA P-256 rather than ML-DSA, with the `renew --rotate-ca` that replaces it); `ca.internal.<credential\|access>` for auth's internal CAs from its trust bundle while it runs (a warning within a year of expiry), and `ca.internal.<credential\|access>.key` when one of them still has a classical key (rotate the one that signs, retire an older one still trusted); `ca.auth-shared` (info) when `[auth] ca` is the storage and gateway CA, whose expiry is then auth's; `renew.leftovers.<service>` (a warning for a `.next` or `.previous` file beside a server's certificate or key with no renewal pending; a staged `<key>.next` is what `renew --rekey <service>` uses); and with the web interface `web.tls`: its certificate is there for an `https://` origin and names the origin's host
Services	`cluster`, `service.<name>` (under docker every container of the project with its health check, the web interface's included), `port.<name>` (accepts a connection while running; free while stopped), `gateway.inventory` (fresh from every instance)
Operations	`journal` (an interrupted command, with the command that finishes it), `backups` (info), and under docker `adopt.leftovers`, `adopt.project.<name>` (an old Compose project that still mounts the cluster's data: warned while stopped, failed while it runs) and `adopt.volume.<name>` (the old web database volume adopt copied)

cluster counts every service the cluster runs as the runtime lists it: under docker with the web interface, every container of the project, Scribe's and the web interface's apart (running (16 services: 3 Scribe, 13 web)). A stopped cluster warns (cluster and each service.<name>, fixed by up); a port another process holds, a service that exited or reports unhealthy, or a gateway without fresh inventory fails. A definition that does not read, or a cluster that cannot be selected, is a failed definition check, not an error: doctor always reports.

With a remote cluster (--remote or [remote]), this machine checks its tools and ssh, probes the remote as every remote command does, and runs the remote's own scribe-ctl doctor --json in the cluster's directory; its sections join the report after this machine's, marked on HOST:DIR, with one summary and one exit code. Nothing is pushed or installed: a remote without a scribe-ctl that answers is a failed remote.ctl check naming the command that installs one, and one of another release is reported (remote.ctl warns) and still asked. The only thing doctor creates is ssh's control socket, in a private directory under /tmp removed when it ends.

A cluster on another machine (its definition has a [remote] table):

text

$ scribe-ctl --config ./scribe-cluster.toml doctor
Scribe · local · doctor · native runtime · on deploy@scribe-host:scribe
  Definition  /home/me/scribe/scribe-cluster.toml
  Read only: nothing is written, locked, started or stopped.

  …

  SSH AND THE REMOTE
    ok    ssh.reach          deploy@scribe-host:scribe answered and logged in within 0.03 s
    ok    ssh.login          key login in batch mode: no password or host-key prompt
    ok    ssh.shared         later commands share the one connection (ControlMaster)
    ok    remote.host        macos/aarch64; rsync 3 no; docker no
    ok    remote.directory   scribe exists
    ok    remote.definition  scribe/scribe-cluster.toml holds cluster local as this machine's definition does
    ok    remote.ctl         bin/0.1.0/scribe-ctl is this release, 0.1.0
    ok    remote.doctor      the remote's scribe-ctl 0.1.0 checked the cluster there

  …

  SERVICES · on deploy@scribe-host:scribe
    ok    cluster            running (1 service)
    ok    service.storage-a  running, pid 75477, tcp://127.0.0.1:47547
    ok    port.storage-a     127.0.0.1:47547 accepts connections (0.2 ms)

  …

  Healthy · 27 ok · 1 info · 0 warn · 0 fail · exit 0

remote.ctl is ok when the remote runs the scribe-ctl every remote command from here would run (the rule of Remote operation): this release exactly, or, from a machine of another OS or architecture, the one bin/controller or bin/current names. Its fix names what actually works from here. On the remote's own platform, any command run with --remote installs this machine's build (as does bootstrap --binaries DIR with a release directory of it); from another platform only bootstrap can, and it does not touch the cluster:

text

  SSH AND THE REMOTE
    …
    FAIL  remote.ctl         no scribe-ctl of this release answers in scribe/bin
                             fix: this machine's scribe-ctl cannot run on the remote (linux/x86_64, this is macos/aarch64): `scribe-ctl --config /home/me/scribe/scribe-cluster.toml bootstrap --source <checkout>` builds this release's there in Docker from a checkout of 0.1.0, without touching the cluster; doctor pushes nothing

A remote without docker gets bootstrap --binaries DIR (this release built for the remote) instead. The same fix follows a remote.doctor failure, a remote scribe-ctl too old to know doctor.

Exit status: 0 when every check is ok or info, 1 when the worst is a warning, 3 when any check fails. 2 stays clap's usage error, as for every command, so a scheduler can tell a sick cluster from a mistyped command. --json prints one document (see JSON output); expiry checks add expires with at and days_left, so a monitor needs no text parsing:

scribe-ctl --config ./scribe-cluster.toml --json doctor \
  | jq -r '.sections[].checks[] | select(.expires) | "\(.expires.days_left)\t\(.id)"' | sort -n

A healthy cluster:

text

$ scribe-ctl --config ./scribe-cluster.toml doctor
Scribe · local · doctor · native runtime
  Definition  /srv/scribe/scribe-cluster.toml
  Read only: nothing is written, locked, started or stopped.

  CONTROLLER
    ok    ctl               scribe-ctl 0.1.0 for macos/aarch64
    ok    definition        /srv/scribe/scribe-cluster.toml defines cluster local (gateway variant) under runtime native
    ok    definition.files  all 18 files and directories it names exist

  BINARIES
    ok    binaries.active      0.1.0 for macos/aarch64 in cluster-state/local/bin/0.1.0
    ok    binaries.controller  this scribe-ctl is the cluster's release, 0.1.0
    ok    binaries.installed   1 set installed

  DISK SPACE
    ok    disk.backups    room for an upgrade's backups: 1.18 MiB to copy to cluster-state/local/backups, 422.20 GiB available
    ok    disk.auth       422.20 GiB available where data/auth holds 1.01 MiB
    ok    disk.storage-a  422.20 GiB available where data/storage-a holds 177.33 KiB

  DATA
    ok    data.auth       data/auth: auth_schema 4, as the active scribe-auth reads and writes
    ok    data.storage-a  data/storage-a: catalog_schema 1, envelope 1, object_store 1, repository_schema 3, as the active scribe-server reads and writes
    ok    data.gateway    data/gateway: route_store_schema 2, as the active scribe-gateway reads and writes

  SECRETS AND PERMISSIONS
    ok    permissions.auth-root  data/auth is reachable by its owner only
    ok    permissions.secrets    10 secret files are private to their owner
    ok    auth.witness           secrets/auth.revision, outside auth's data root
    ok    auth.pepper            secrets/auth.pepper, outside auth's data root

  CERTIFICATES AND CREDENTIALS
    ok    expiry.administrator-credential     administrator credential (secrets/admin.credential): does not expire
    ok    expiry.auth-certificate             auth certificate (pki/auth/server.pem): expires in 364 days (2027-09-27)
    ok    expiry.storage-a-server-credential  storage-a server credential (secrets/storage-a.credential): expires in 364 days (2027-09-27)
    ok    expiry.inventory-credential         inventory credential (secrets/monitor.credential): expires in 364 days (2027-09-27)
    ok    expiry.storage-a-certificate        storage-a certificate (pki/storage-a/server.pem): expires in 364 days (2027-09-27)
    ok    expiry.gateway-credential           gateway credential (secrets/gateway.credential): expires in 364 days (2027-09-27)
    ok    expiry.gateway-certificate          gateway certificate (pki/gateway/server.pem): expires in 364 days (2027-09-27)
    ok    expiry.storage-and-gateway-ca       storage and gateway CA (pki/ca/ca.pem): expires in 3649 days (2036-09-24)
    ok    expiry.auth-ca                      auth CA (pki/auth-ca/ca.pem): expires in 3649 days (2036-09-24)

  SERVICES
    ok    cluster            running (3 services)
    ok    service.auth       running, pid 70911, tls://127.0.0.1:47443
    ok    service.storage-a  running, pid 70943, tls://127.0.0.1:47447
    ok    service.gateway    running, pid 70960, tls://127.0.0.1:47557
    ok    port.auth          127.0.0.1:47443 accepts connections (0.2 ms)
    ok    port.storage-a     127.0.0.1:47447 accepts connections (0.1 ms)
    ok    port.gateway       127.0.0.1:47557 accepts connections (0.1 ms)
    ok    gateway.inventory  the gateway has fresh inventory from every instance

  OPERATIONS
    ok    journal  no interrupted command
    ok    backups  no upgrade backups kept

  Healthy · 35 ok · 0 info · 0 warn · 0 fail · exit 0

The same cluster stopped, with auth's data root opened to other accounts:

text

$ scribe-ctl --config ./scribe-cluster.toml doctor
  …

  SECRETS AND PERMISSIONS
    FAIL  permissions.auth-root  /srv/scribe/data/auth is accessible to other accounts (mode 755) and holds private keys; restrict it with chmod 700 /srv/scribe/data/auth
                                 fix: chmod 700 /srv/scribe/data/auth
    ok    permissions.secrets    10 secret files are private to their owner
    ok    auth.witness           secrets/auth.revision, outside auth's data root
    ok    auth.pepper            secrets/auth.pepper, outside auth's data root

  SERVICES
    WARN  cluster            stopped
                             fix: `scribe-ctl --config /srv/scribe/scribe-cluster.toml up` starts it
    WARN  service.auth       stopped
                             fix: `scribe-ctl --config /srv/scribe/scribe-cluster.toml up` starts the cluster
    WARN  service.storage-a  stopped
                             fix: `scribe-ctl --config /srv/scribe/scribe-cluster.toml up` starts the cluster
    WARN  service.gateway    stopped
                             fix: `scribe-ctl --config /srv/scribe/scribe-cluster.toml up` starts the cluster
    ok    port.auth          127.0.0.1:47443 is free for the next start
    ok    port.storage-a     127.0.0.1:47447 is free for the next start
    ok    port.gateway       127.0.0.1:47557 is free for the next start

  …

  Failed checks: see the fixes above · 29 ok · 0 info · 4 warn · 1 fail · exit 3

df

Show live disk usage for every storage server (apparent bytes)

text

scribe-ctl df [OPTIONS]

Alias: usage. No command-specific options. Select --cluster NAME when multiple clusters are defined.

repo list

List every repository with its storage server, head and size

text

scribe-ctl repo list [OPTIONS]

No command-specific options. Select --cluster NAME when multiple clusters are defined. Repository names are grouped by storage server; equal names on different servers remain separate entries.

Live inventory behavior

All three commands query the existing paginated storage inventory API. They include configured storage services and additional gateway backends. An exact endpoint match between a storage service and gateway backend is queried once, under the storage service name. The gateway backend's CA is reused for that match unless storage.ca overrides it. The gateway need not be running. Port-zero local stores use their recorded endpoint only while their data-root lock is held.

Authenticated storage needs storage.monitor_credential, the credential file of a service account with a namespace reader grant in the store's namespace, and trusts storage.ca, the matching gateway backend's CA, or certificates.ca_dir/ca.pem. The store's server credential is not a user credential and is never used for inventory. An authenticated gateway backend with no matching storage service has no monitor credential and is reported as a per-server error. Inventory never provisions credentials or changes permissions. See the cluster guide for configuration.

Terminal inventory uses aligned tables and readable IEC sizes (KiB, MiB, GiB, and larger units). Use --json for automation: size fields remain exact integer byte counts, with no rounding or unit suffixes.

Used bytes are apparent data-root sizes, sampled live rather than an atomic cluster snapshot. Available bytes are shown per server and never added together: several roots can share a filesystem. Totals cover responding configured targets; alias endpoints pointing at the same store should not be configured twice. This is the online inventory, not the detailed offline scribe-server usage object breakdown.

Queries use up to four connections at once per cluster and a 10-second connect/I/O timeout (the gateway's backend_timeout_secs when configured). Every page is checked for ordered repository IDs, stable instance identity and repository count. Retry if repositories change during collection.

Unreachable, unauthorized or invalid responses appear as per-server errors; successful results remain visible, totals are marked partial, and exit status is 1. JSON emits one result with ok, command, cluster, and clusters. Each cluster contains servers, totals, and lifecycle (only populated for status --storage). Failed servers have null size fields and an error.

logs

Show the last lines of a service or supervisor log

text

scribe-ctl logs [OPTIONS] <SERVICE>

Argument or option	Meaning
`<SERVICE>`	Service name (auth, gateway, or a storage name), or supervisor; under docker any service of the Compose project (the web interface's too)
`--lines <LINES>`	Number of lines, reading at most the final 64 KiB [default: 40]

--lines defaults to 40, accepts 1–1000, and reads at most the final 64 KiB. SERVICE is auth, gateway, a configured storage name, or supervisor. This prints a bounded tail; it does not follow the log. Under docker it is docker compose logs --no-log-prefix --tail LINES SERVICE, for any service of the project (api, worker, … of the web interface too).

ssh

Open a shell in the remote cluster's directory, or run a command there (after --). Needs --remote or a [remote] table

text

scribe-ctl ssh [OPTIONS] [-- <COMMAND>...]

Argument or option	Meaning
`[COMMAND]...`	The command and its arguments, each passed as one word

Without a command it opens a login shell (with a terminal) in the cluster's directory on the remote; there, bin/controller names the newest scribe-ctl installed (bin/<build>/scribe-ctl --config <file> status runs the cluster's commands directly, on the copy of the definition). With one, each word reaches the remote command exactly as given (quoted for the remote shell), in that directory, and its exit code is this command's. Nothing is pushed or installed.

text

$ scribe-ctl --config ./scribe-cluster.toml ssh -- ls
CLUSTER-INFO.txt
bin
cluster-state
data
pki
scribe-cluster.toml
secrets
$ scribe-ctl --config ./scribe-cluster.toml ssh -- sh -c 'exit 7'; echo "exit $?"
exit 7

bootstrap

Install or refresh the scribe-ctl that runs this cluster's commands on its remote machine, without touching the cluster: built there in Docker from a checkout (--source; for a remote of another OS or architecture, which this machine's scribe-ctl cannot run on) or from a release directory built for the remote (--binaries). It is named in bin/controller, which later commands from a machine whose own build cannot run there use; bin/current, the definition there, services, images, lock and journal stay as they are, so it is safe beside a command running there. Needs --remote or a [remote] table

text

scribe-ctl bootstrap [OPTIONS]

Argument or option	Meaning
`--source <DIR>`	A Scribe checkout of this scribe-ctl's version: what its scribe-ctl build reads is pushed to `upload/controller/source` there and built in Docker (`docker/Dockerfile.ctl`) as build `<version>-src.<hash of the pushed files>`. Needs docker on the remote. Nothing is pushed or built when that build is there already
`--binaries <DIR>`	A release directory of this scribe-ctl's version built for the remote's OS and architecture: its executables are pushed to `upload/controller/binaries` and installed as `bin/<build>/` there
`--plan`	Show what would be pushed, built and installed, after probing the remote, without changing anything there

A scribe-ctl here can only install its own build on a remote of the same OS and architecture, which every remote command does by itself. From a laptop of another platform (macOS, say, and a Linux server), the remote runs the scribe-ctl the last --source command built there, which knows only the commands of its checkout. bootstrap --source brings a newer one over without an upgrade: it pushes the checkout (the same selection as any --source push, into upload/controller/source, so the upload/source of a command running there is left alone), builds only scribe-ctl there, installs it as bin/<build>/ and names it in bin/controller. Nothing else changes: not bin/current, the definition there, the cluster's binary set or images, its services, lock or journal, so it takes no lock and may run while an upgrade runs there (an existing bin/<build> that answers as that build is never replaced). Rules 3 and 4 of Remote operation then pick it for every command from a machine whose own build cannot run there, doctor included; a machine of the remote's platform keeps installing and running its own build. A later --source setup, upgrade or adopt still names its own build in bin/current and bin/controller.

text

$ scribe-ctl --config ./scribe-cluster.toml --remote deploy@scribe-host:scribe bootstrap --source ../scribe --plan
  Checking deploy@scribe-host:scribe over ssh…
Scribe · deploy@scribe-host:scribe · Controller plan
  Would push the checkout ../scribe to scribe/upload/controller/source (1812 entries, 13.0 MB)
  Would build scribe-ctl 0.1.0 (build 0.1.0-src.25b366157b78) for linux/x86_64 there in Docker (docker/Dockerfile.ctl)
  Would install it as bin/0.1.0-src.25b366157b78/scribe-ctl, named in bin/controller
  Would leave the cluster as it is: bin/current (0.1.0 (build 0.1.0-src.3fa9c2e1b7d0)), its services, images, lock and journal

  Plan only: nothing was pushed, built or installed.
$ scribe-ctl --config ./scribe-cluster.toml --remote deploy@scribe-host:scribe bootstrap --source ../scribe
  Checking deploy@scribe-host:scribe over ssh…
  Pushing the checkout ../scribe to scribe/upload/controller/source (1812 entries, 13.0 MB); tar over ssh…
  Building scribe-ctl 0.1.0 (build 0.1.0-src.25b366157b78) for linux/x86_64 there in Docker (docker/Dockerfile.ctl); the first build takes a while…
Scribe · deploy@scribe-host:scribe · Controller installed
  Pushed the checkout ../scribe to scribe/upload/controller/source (1812 entries, 13.0 MB)
  Built scribe-ctl 0.1.0 (build 0.1.0-src.25b366157b78) for linux/x86_64 there in Docker (docker/Dockerfile.ctl)
  Installed it as bin/0.1.0-src.25b366157b78/scribe-ctl, named in bin/controller
  Left the cluster as it is: bin/current (0.1.0 (build 0.1.0-src.3fa9c2e1b7d0)), its services, images, lock and journal

  Remote commands from a machine whose own scribe-ctl cannot run on linux/x86_64 now run bin/0.1.0-src.25b366157b78/scribe-ctl.

Run again with the same checkout, it pushes and builds nothing (Controller unchanged, bin/<build>/scribe-ctl is there already). --binaries DIR pushes a release directory built for the remote instead (checked there: its scribe-ctl must answer --build-info for the remote's OS and architecture and this version) and installs its executables. Both refuse another version than this scribe-ctl's, since remote runs from here only pick this version. Locally, without --remote or a [remote] table, it is refused. --json returns remote, plan, build, target, controller, unchanged, current and steps.

Internal supervisor entry point

up --daemon launches the hidden supervise action. It is an implementation entry point; use up for normal operation so readiness and lifecycle state are coordinated by the controller.

text

scribe-ctl supervise [OPTIONS] --generation <GENERATION>

Argument or option	Meaning
`--generation <GENERATION>`	Supervisor run identifier generated by `up`, matching readiness to this run

The global configuration and cluster selectors apply here too. This action is intentionally omitted from the public help command list.

Use cases

Set up a new cluster

scribe-ctl --config ./scribe-cluster.toml setup --variant gateway --plan
scribe-ctl --config ./scribe-cluster.toml setup --variant gateway
scribe-ctl --config ./scribe-cluster.toml where
scribe-ctl --config ./scribe-cluster.toml up

Grow a storage cluster

scribe-ctl --config ./scribe-cluster.toml down
scribe-ctl --config ./scribe-cluster.toml add auth --plan
scribe-ctl --config ./scribe-cluster.toml add auth
scribe-ctl --config ./scribe-cluster.toml add gateway
scribe-ctl --config ./scribe-cluster.toml up

Upgrade to a new release

scribe-ctl --config ./scribe-cluster.toml upgrade --binaries /opt/scribe/0.2.0 --plan
scribe-ctl --config ./scribe-cluster.toml upgrade --binaries /opt/scribe/0.2.0
scribe-ctl --config ./scribe-cluster.toml status

Run a cluster in Docker, with the web interface

# from the checkout: build and install scribe-ctl, then set up from the same checkout
./scripts/install.sh --setup gateway --config ~/scribe/scribe-cluster.toml -- --runtime docker --web --source . --plan
./scripts/install.sh --setup gateway --config ~/scribe/scribe-cluster.toml -- --runtime docker --web --source .
scribe-ctl --config ~/scribe/scribe-cluster.toml up
# later, after pulling new source (the build is named by the commit, or SCRIBE_BUILD_ID):
./scripts/install.sh --upgrade --config ~/scribe/scribe-cluster.toml -- --source . --plan
./scripts/install.sh --upgrade --config ~/scribe/scribe-cluster.toml -- --source .

Renew credentials before they expire

scribe-ctl --config ./scribe-cluster.toml renew --plan
scribe-ctl --config ./scribe-cluster.toml renew
# from a scheduler, daily; a no-op until 30 days before an expiry
scribe-ctl --config /srv/scribe/scribe-cluster.toml renew --json

Take over an existing Docker deployment

scribe-ctl --config ./scribe-cluster.toml adopt --runtime docker --admin-credential /srv/old-scribe/operator/admin.credential --source . --plan
scribe-ctl --config ./scribe-cluster.toml adopt --runtime docker --admin-credential /srv/old-scribe/operator/admin.credential --source .
scribe-ctl --config ./scribe-cluster.toml status

Operate a cluster on another machine

scribe-ctl --config ./prod.toml --cluster prod --remote deploy@scribe-host:/srv/scribe setup --variant gateway --binaries ./linux-x86_64 --plan
scribe-ctl --config ./prod.toml --cluster prod --remote deploy@scribe-host:/srv/scribe setup --variant gateway --binaries ./linux-x86_64
scribe-ctl --config ./prod.toml up
scribe-ctl --config ./prod.toml status
scribe-ctl --config ./prod.toml upgrade --binaries ./linux-x86_64-next
scribe-ctl --config ./prod.toml renew
scribe-ctl --config ./prod.toml ssh
# from a laptop of another platform: refresh the scribe-ctl there, nothing else
scribe-ctl --config ./prod.toml bootstrap --source .

Inspect a managed cluster

scribe-ctl --config ./scribe-cluster.toml doctor
scribe-ctl --config ./scribe-cluster.toml status --storage --json
scribe-ctl --config ./scribe-cluster.toml df
scribe-ctl --config ./scribe-cluster.toml repo list
scribe-ctl --config ./scribe-cluster.toml access
scribe-ctl --config ./scribe-cluster.toml logs supervisor --lines 50

Add --cluster NAME when selection is required. For a guided walk through setup, add, and everyday operation, follow the tutorials (4. Operate the cluster covers these commands). The cluster guide covers configuration and lifecycle rules.

Output and exit codes

doctor exits 0 when every check passes, 1 with warnings only and 3 with a failed check. Otherwise, exit 0 means success, 1 means runtime failure (including a setup --plan, add … --plan, upgrade --plan, renew --plan or adopt --plan whose checks fail, and an upgrade, renewal or adoption that was rolled back), and 2 means invalid CLI arguments. renew with nothing due exits 0. An upgrade handed over to a newer scribe-ctl exits with that controller's exit code, a remote command with the remote scribe-ctl's (255 when ssh itself failed), and ssh -- COMMAND with the command's. Long-running foreground commands report their final JSON result after shutdown.

--json selects machine-readable stdout results; diagnostics stay separate. See JSON output for schemas, help, and error behavior, and logging for filters, sinks, timing events, and build features.