Local clusters with scribe-ctl

For every controller command and option, see the controller CLI reference. This guide covers the cluster definition, setup, and service lifecycle workflows.

All command-line tools support global --json; see JSON output for automation for results, errors, and exit codes.

scribe-ctl sets up and supervises named clusters from one definition file. Each cluster has one optional auth service, up to 256 storage services, and an optional gateway, run as supervised local processes, or as containers of one Docker Compose project with the web interface beside them (Run a cluster in Docker), on this machine or on another one over ssh (Remote operation). Certificate generation is built in; no external certificate tool is required. renew keeps the service credentials valid (Renew service credentials).

New to it? The tutorials build a cluster step by step: one store, then auth, a gateway with a second store, everyday operation and an upgrade, and the web interface in Docker.

Set up a new cluster

Install the executables first: from a source checkout, scripts/install.sh (scripts/install.ps1 on Windows) builds and installs them, and with --setup VARIANT --config FILE (-Setup VARIANT -Config FILE) hands over to scribe-ctl setup in the same line (see build and install). Pick a variant, look at the plan, then run it:

scribe-ctl --config ./scribe-cluster.toml setup --variant gateway --plan
scribe-ctl --config ./scribe-cluster.toml setup --variant gateway
scribe-ctl --config ./scribe-cluster.toml where
scribe-ctl --config ./scribe-cluster.toml up

setup writes the definition (named cluster local, every path relative to the file, services on loopback), then creates everything the cluster needs: data roots, both server-certificate CAs (pki/ca for storage and gateway, pki/auth-ca for auth), server certificates, the auth pepper and revision witness, the bootstrap administrator credential, each storage server's credential, the gateway's scribe-gateway service account with its grants and credential, and a scribe-monitor credential that df and repo list read inventory with. --namespace NAME sets the authenticated server's namespace (default studio); --port-offset N moves every default port by N.

Setup also records the executables it finds (beside scribe-ctl, or where the definition's [binaries] names them) as the cluster's first binary set: it checks they are one release for this machine, copies them into <state_dir>/bin/<build>/, and names that set in <state_dir>/bin/current. Services start from the set current names, so a later upgrade can switch releases and switch back.

Last, setup writes CLUSTER-INFO.txt beside the definition. It lists every file and what it is for, which ones to back up, which to give clients, which to keep offline, the endpoints, where the web interface's TLS certificate goes, and the exact next commands, such as adding a person and logging in. Secrets appear there only as paths. scribe-ctl where prints the same summary, current, at any time; see the reference for an example.

--plan runs every check for real (definition, binaries, permissions of existing secrets, the accounts file, whether the cluster runs, free listen ports, certificate and data-root consistency) and prints the steps with the reason for each. It writes nothing, takes no lock and starts nothing. Setup executes exactly that plan, planned again under its lock.

To change ports, names or paths before anything exists, write the definition yourself (start from scribe-ctl example or edit the one setup --plan describes) and run setup without --variant: with an existing definition, setup uses it as written. A --variant it does not match is refused.

Rerun, resume, and what setup refuses

Setup is idempotent: a rerun on a complete cluster changes no file it made. It still starts auth briefly to confirm registrations, grants and credentials. It never replaces a credential, also one close to expiry: its plan points to renew for those.

Every step is recorded in state_dir/journal.json before and after it runs. An interrupted setup (a crash, a killed process, a full disk) resumes when run again: finished steps are not repeated, and a step that had started is redone after its partial output is moved aside as <name>.interrupted-<run> beside the original, never deleted. If the interruption left the temporary auth running, setup refuses until scribe-ctl down drains it. The journal is removed when a run completes.

State that setup did not make is refused, not replaced: a partial PEM pair, a CA directory without its key (bring the key back from offline storage), a data root that holds files but no format marker, an auth pepper or witness without its database, or a generated file with other contents. The message says what was found and what to do. Setup never regenerates a pepper or revision witness for an existing auth database.

While the cluster runs, setup performs only its identity steps, through the running auth (for example, accounts added to accounts_file), and refuses steps that need the cluster stopped.

Grow a cluster: add auth, then a gateway

A cluster that started as the storage variant gains auth, and an auth cluster gains a gateway, without losing a repository:

scribe-ctl --config ./scribe-cluster.toml down
scribe-ctl --config ./scribe-cluster.toml add auth --plan
scribe-ctl --config ./scribe-cluster.toml add auth
scribe-ctl --config ./scribe-cluster.toml add gateway
scribe-ctl --config ./scribe-cluster.toml up

Each add rewrites the definition canonically (the previous file is kept as <file>.bak; --plan shows the diff, and for add gateway the new scribe-gateway.toml), then provisions the result exactly as setup would, with the same checks, journal and resume. add auth puts auth on 127.0.0.1:7443 (moved by as much as the store is from 7447; --listen chooses), creates both CAs if missing, issues the auth and store certificates, initializes auth with its pepper, revision witness and administrator credential, binds every store to its namespace (--namespace, default studio) in place, registers it and issues its credential, and provisions the scribe-monitor inventory account. add gateway writes the gateway's configuration (one instance per store), issues its certificate, initializes its ledger, and provisions the scribe-gateway account with reader and delegate grants and its credential.

Data roots are never re-initialized or deleted: the store keeps its repositories, history and IDs, and a gateway imports them on its first inventory poll (as studio/NAME). The cluster must be stopped for either add; while it runs, the plan's cluster check fails and nothing changes. A bound store no longer starts without auth.

After add auth, clients use tls:// for the store with --tls-ca pki/ca/ca.pem and log in through auth with --auth-ca pki/auth-ca/ca.pem; plaintext connections are refused. Nobody has access until an administrator adds people and grants (the Next commands in CLUSTER-INFO.txt). A workspace created before auth has no signed-in owner: it still syncs, but submitting needs a new workspace created after logging in. After add gateway, clients use the gateway's address and log in once more for it, since a saved session belongs to the server it was made for.

An add that finds its service complete changes nothing and says so. It refuses a cluster that is not its source variant (add gateway on storage only: run add auth first; a custom topology: change the definition by hand), options that shape a service that already exists, and a service that exists but is incomplete with no interrupted add to resume (run setup to complete it). See the reference for a full plan.

Upgrade to a new release

Unpack the new release into a directory of its own, look at the plan, then run it with the controller you have:

scribe-ctl --config ./scribe-cluster.toml upgrade --binaries /opt/scribe/0.2.0 --plan
scribe-ctl --config ./scribe-cluster.toml upgrade --binaries /opt/scribe/0.2.0

The release's own scribe-ctl runs its upgrade, because only it knows that release's migrations: when the directory holds a newer one, yours hands the command over to it and reports its result. The plan checks that the directory is one complete release for this machine (every executable answers --build-info), that it is not older than the active set, that no data root records a format newer than the new executables read, and which formats change. With the cluster stopped it also runs scribe-auth migrate --dry-run and scribe-server migrate --dry-run with the new executables. It measures the roots it will back up and fails its free space check unless the backups' filesystem has room for them plus a reserve of a tenth of their size, at least 1 GiB. --plan writes nothing, takes no lock and touches no service.

The upgrade then installs the new set side by side in <state_dir>/bin/<build>/ while the cluster still serves, drains it, backs up each root a migration will change (auth, stores) to <state_dir>/backups/<stamp>/, switches bin/current, migrates with the new executables (auth first, then each store), starts the cluster and waits until every service is ready. A cluster that was stopped is stopped again afterwards. Downtime runs from the drain to the health check.

Auth's revision witness lives outside its data root and its backups, so that a restored old copy cannot revive revoked credentials unnoticed. The upgrade keeps the witness that matches auth's backup in its journal instead, and a rollback puts both back together; once the upgrade completes that copy is gone with the journal.

If anything fails once a running cluster is drained (for a stopped one, from the switch on), the upgrade rolls back by itself: it stops what runs, restores each migrated root from its backup (keeping the migrated one beside it as <root>.upgraded-<stamp>; auth with its witness), switches back to the previous set and starts it if the cluster was running. A failed backup therefore leaves the old release serving, not a drained cluster. Each backup records a BLAKE3 digest of every file; the copy is checked against them right away, and a backup that no longer matches is refused before a restore moves anything. A crash at any point is resumed by running the same command again. Nothing is deleted: earlier sets and backups stay until you prune them, and where lists them. status names the active set; use the new release's scribe-ctl (<state_dir>/bin/<build>/scribe-ctl, or a copy of it) from then on.

Two migrations are refused with what to do by hand: auth's schema 3 to 4, which replaces every credential (upgrading the auth data directory), and any change of the gateway ledger, which has no migrate command. See the reference for every check, a real plan and a rollback. The same sequence for a deployment scribe-ctl does not run is in upgrade a deployment by hand.

Prune old releases

Every upgrade keeps the release it replaced, so each one leaves a binary set behind, and under Docker its images. prune removes what old releases left, and only what scribe-ctl made:

scribe-ctl --config ./scribe-cluster.toml prune --plan
scribe-ctl --config ./scribe-cluster.toml prune            # --keep 1: the active set and the one before it stay

It removes binary sets other than the active one and the --keep newest before it, which stay for a rollback. Under Docker it also removes the images tagged with those sets' builds (scribe/*:<build>) that no container uses; an image a container still uses stays, and the plan says which. It removes the scribe-ctl builds a remote command installed in the cluster's directory (bin/<build>/scribe-ctl) that no kept set, bin/controller or bin/current names, and upgrade backups older than the --keep newest.

Docker's build cache, other images, volumes and containers belong to the machine and are never touched, since other work on it may use them. doctor warns when the build cache grows (docker.build-cache), and docker builder prune frees it when you choose. renew never prunes; you run prune when you choose, with --remote for a cluster on another machine as any command.

Run a cluster in Docker

With --runtime docker, setup writes runtime = "docker" and every service runs as a container of one Compose project that scribe-ctl generates and drives with the docker CLI. You need Docker Engine with Compose v2 and a Scribe checkout of the same version as your scribe-ctl: images are built from it and tagged with that scribe-ctl's build.

./scripts/install.sh --setup gateway --config ~/scribe/scribe-cluster.toml -- --runtime docker --web --source .
scribe-ctl --config ~/scribe/scribe-cluster.toml up
scribe-ctl --config ~/scribe/scribe-cluster.toml where

(or, with scribe-ctl already installed from this checkout, scribe-ctl --config ~/scribe/scribe-cluster.toml setup --variant gateway --runtime docker --web --source .). Tutorial 5 walks through it with real output.

Everything else is as natively: the same definition (with a fixed port for every service and auth's pepper and witness in a directory of their own, secrets/auth/), the same data roots on the host, bind-mounted into the containers at fixed paths, the same plans, journal, backups and rollback. Containers reach each other by service name on the project network (auth, each store's name, gateway) and publish their ports on the definition's addresses only. Each Scribe service's container runs as the owner of the definition's directory, with a read-only root filesystem, no capabilities and no privilege escalation, and mounts only its own data root and files (a serving process never sees a CA key or the administrator credential). scribe-ctl itself runs on the host and provisions identities through auth's published port. up is docker compose up --wait, down is docker compose stop, logs SERVICE shows a container's log, and where prints the docker compose command for the project if you want to look yourself. The generated files in <state_dir>/docker/ are rewritten before every start: change the definition instead.

--web adds the web interface (apps/web) to the same project, with its database in data/web, its secrets in secrets/web/, and, with auth, the scribe-web service account and its reader credential. It is published on http://localhost:8080. Its TLS certificate is yours: put fullchain.pem and key.pem in secrets/web-tls/, set [web] origin to the https:// address, and the next up serves HTTPS; see the web interface guide.

To upgrade, update the checkout and let the install script build the new scribe-ctl and hand the checkout to it (it names the build by the commit, so two builds of one version differ; set SCRIBE_BUILD_ID to choose the name):

./scripts/install.sh --upgrade --config ~/scribe/scribe-cluster.toml -- --source . --plan
./scripts/install.sh --upgrade --config ~/scribe/scribe-cluster.toml -- --source .

It builds the new images first (with --plan too), then upgrades as natively, switching the project to the new images; with the web interface it also backs up the web database and runs the new worker's migrate.js, whose dry run against the running database refuses the upgrade before anything stops. The reference has the details and real plans. Setup never moves a cluster between runtimes: an existing native cluster keeps running natively (its data roots and credentials name host addresses), and a Docker cluster starts from a new definition.

Renew service credentials

Setup issues every service credential for 365 days: each authenticated store's server credential, the gateway's (scribe-gateway), the inventory credential (scribe-monitor) and, under docker with the web interface, the web reader's (scribe-web); and each service's TLS server certificate for 365 days. scribe-ctl renew replaces each credential, the administrator credential and each server certificate that expires within 30 days, while the cluster runs. Only the web interface's own certificate (fullchain.pem in [web] tls_dir) stays yours to renew:

scribe-ctl --config ./scribe-cluster.toml renew --plan   # what is due, and the steps
scribe-ctl --config ./scribe-cluster.toml renew

It issues the replacement beside the file (<file>.next), keeps the current file as <file>.previous, puts the new one in place, restarts the services that read their credential only at startup (the stores and the gateway, one at a time while the rest keep serving: natively the supervisor restarts them in place, under docker their containers are recreated), and proves each new credential for its purpose: a store serves a namespace read it authorizes with its new credential, the gateway polls every instance, and the inventory and web reader credentials read every namespace. Only then is each old credential revoked. If a proof fails, the old credentials go back in place, the holders restart with them, the new ones are revoked, and renew exits 1: the cluster keeps working with the old credentials until their own expiry, and the next run tries again. A stopped cluster is started for the proofs and stopped again. An interrupted renewal resumes when run again (it adopts a credential auth already issued rather than issuing a second). The administrator credential is renewed the same way (for 10 years, proved with an administrator request); the bootstrap one setup writes never expires, so it is never due.

A server certificate is reissued for the key it already has, by the CA that signed it, and followed in its file by that CA (clients fingerprint the top of the chain a server presents). A reissue, a rekey and a rotation keep every name the certificate carries and add localhost, loopback, a container's service name and the names the definition lists (tls.dns/tls.ip, and for the gateway dns/ip in its table), so no name clients connect by is ever dropped. A certificate file that holds its certificate alone (set up by hand, or taken over) makes its server present no CA, so clients that do not trust it yet cannot discover it: doctor warns (chain.<service>) and a plain renew writes the file again followed by its CA, the same certificate and key. Only the service it belongs to restarts, and a TLS handshake trusting the CA clients hold proves it before the kept copy is removed; clients notice nothing. The CA's key must be where renew runs: beside the cluster (ca-key.pem in certificates.ca_dir or auth_ca_dir, as setup writes it), or in a directory you name with --ca-dir DIR (the storage and gateway CA) or --auth-ca-dir DIR (the auth CA). When it is neither, the plan fails for that certificate and names the command. With --remote, the directory is on the machine you run scribe-ctl on: the certificates are signed there and only they travel to the cluster; the key never leaves. Details and real output: renew.

status and where warn 30 days ahead (a CA a year ahead) and name the command. Better, run renew from a scheduler once a day: it renews what is due and retires CAs whose overlap ended (a rotation's old CA, auth's older internal CAs), and is a no-op (exit 0, nothing written, about a second) until something is due. It never prompts and exits 1 whenever it could not finish, always leaving working credentials in place. scribe-ctl installs no schedule itself: set one up with your system's scheduler, as the account that owns the cluster's files, and keep its output (progress on stderr, the result on stdout, or --json).

Schedule renew on Linux

cron, daily at 03:17:

17 3 * * * /opt/scribe/scribe-ctl --config /srv/scribe/scribe-cluster.toml renew >> /srv/scribe/renew.log 2>&1

systemd, as a oneshot service and a timer. Put both files in place, then systemctl daemon-reload && systemctl enable --now scribe-renew.timer; systemctl list-timers scribe-renew.timer shows the next run and journalctl -u scribe-renew the last one's output:

# /etc/systemd/system/scribe-renew.service
[Unit]
Description=Scribe: renew what is due, retire CAs whose overlap ended
# A docker-runtime cluster: its containers must be up.
After=docker.service network-online.target
Wants=network-online.target

[Service]
Type=oneshot
User=scribe
Group=scribe
WorkingDirectory=/srv/scribe
ExecStart=/opt/scribe/scribe-ctl --config /srv/scribe/scribe-cluster.toml renew

# /etc/systemd/system/scribe-renew.timer
[Unit]
Description=Daily Scribe renew

[Timer]
OnCalendar=daily
RandomizedDelaySec=1h
Persistent=true

[Install]
WantedBy=timers.target

Persistent=true runs a missed day's renew at the next boot.

Schedule renew on Windows

Task Scheduler, from an elevated prompt; /ru names the account that owns the cluster's files and /rp * asks for its password once:

schtasks /create /tn "Scribe renew" /sc daily /st 03:17 /ru scribe /rp * ^
  /tr "\"C:\Scribe\scribe-ctl.exe\" --config \"C:\Scribe\scribe-cluster.toml\" renew"

The same from PowerShell, with the output kept in a log:

$action = New-ScheduledTaskAction -Execute 'powershell.exe' -Argument '-NoProfile -Command "& ''C:\Scribe\scribe-ctl.exe'' --config ''C:\Scribe\scribe-cluster.toml'' renew *>> ''C:\Scribe\renew.log''"'
$trigger = New-ScheduledTaskTrigger -Daily -At 3:17am -RandomDelay (New-TimeSpan -Hours 1)
$settings = New-ScheduledTaskSettingsSet -StartWhenAvailable
Register-ScheduledTask -TaskName 'Scribe renew' -Action $action -Trigger $trigger -Settings $settings -User 'scribe' -Password (Read-Host 'Password for scribe')

-StartWhenAvailable runs a missed day's renew once the machine is back. Get-ScheduledTaskInfo 'Scribe renew' shows the last result (0 when it finished).

Schedule renew for a cluster on another machine

Schedule it on the cluster's machine, where no ssh is involved. The scribe-ctl that runs its commands there is bin/<build>/scribe-ctl in the cluster's directory, and bin/controller names the build (each upgrade moves it), so the command reads it on every run; for a cluster in /home/scribe/scribe-ctl:

/bin/sh -c 'exec /home/scribe/scribe-ctl/bin/"$(cat /home/scribe/scribe-ctl/bin/controller)"/scribe-ctl --config /home/scribe/scribe-ctl/scribe-cluster.toml renew'

Use that as the systemd ExecStart= (with WorkingDirectory= the cluster's directory) or as the cron command. Scheduling it here instead, with the definition that has the [remote] table, works too, but then this machine must be on and its ssh key usable unattended from the scheduler's account. A CA key kept on this machine (--ca-dir) is needed only in the months a server certificate is due; the schedule can name it every time.

Auth under the storage and gateway CA

A deployment taken over may have auth's certificate signed by the storage and gateway CA, with [auth] ca naming that CA's ca.pem (doctor says so: ca.auth-shared). That CA then renews and rekeys auth's certificate too, and renew --rotate-ca storage rotates it for auth as well: auth's certificate moves to the new CA with the stores' and the gateway's, and the credentials that carry the CA they trust auth by follow. There is no auth CA of its own to rotate, so --rotate-ca auth is refused with that command.

CA fingerprints for clients

where shows, under each tls:// endpoint, the fingerprint of the CA clients verify it by (CA BLAKE3:…): give it to people with the address, for scribe trust add --trust-fingerprint.

Replace server keys

A plain renew keeps each server's private key. renew --rekey replaces it: for a key you suspect has leaked, or to move servers set up before Scribe generated ML-DSA-65 keys to a post-quantum one. It is never done by a scheduled renew, only by you:

scribe-ctl --config ./scribe-cluster.toml renew --rekey storage-a --plan
scribe-ctl --config ./scribe-cluster.toml renew --rekey storage-a,gateway
scribe-ctl --config ./scribe-cluster.toml renew --rekey          # every TLS server

Each named service (auth, a store with TLS, the gateway) gets a new private key, generated where the cluster runs as <key>.next (only the cluster's account can read it), and a certificate for it from the CA that signs the server now, for the same names; during a rotation's overlap it is followed by the cross certificate, as a reissue is. The new key and certificate go in place together, the old ones kept as <file>.previous; only that service restarts, and a TLS handshake trusting the CA clients hold proves the new pair before the old key is removed. If the proof fails, the old key and certificate go back together and the service restarts with them. Clients trust the CA, not the key: they notice nothing and trust nothing again, and sessions and credentials are unaffected. A name that is not a TLS server of the cluster is refused with the ones that are.

With the CA key on another machine (--remote with --ca-dir DIR), the new key is still made on the cluster's machine and never leaves it: only its public key travels to be certified, and only the certificate comes back. doctor warns about a .next or .previous file left beside a certificate or key with no renewal pending; a staged <key>.next is what the next renew --rekey for that service uses.

Rotate a CA

The two server-certificate CAs (the storage and gateway CA, and the auth CA) are valid for 10 years. status, doctor and renew say so a year before one expires; a rotation replaces it without asking clients to trust anything by hand, as long as they connect during an overlap. It is never started by a scheduled renew, only by you:

scribe-ctl --config ./scribe-cluster.toml renew --rotate-ca storage --plan
scribe-ctl --config ./scribe-cluster.toml renew --rotate-ca storage --overlap-days 90

A rotation makes a new CA and a cross certificate: the new CA's name and key, signed by the old CA and valid for the overlap (90 days by default, never past the old CA). Every server certificate of that CA is reissued by the new CA for the key it has, in one run, and presented followed by the cross certificate, so a client that trusts only the old CA still verifies it, and learns the new CA as it does; a client given the new CA verifies it directly. Before any server changes, the web interface's services (which follow no cross certificate) are restarted trusting both CAs. The new CA then becomes ca.pem (the old one moves to previous/), the trust files the definition names follow, and for the auth CA every credential file ctl manages (servers', the gateway's, the inventory and web reader credentials, the administrator's) is rewritten to carry the new auth CA; the services that read them restart. Each step is proved with TLS handshakes trusting the old and the new CA alone, and a failure puts the old CA back. Give new clients the new ca.pem.

Why server CAs have path length 1, and why an older CA with path length 0 still rotates, is in the four certificate authorities.

status shows the rotation and the day its overlap ends. From that day the scheduled plain renew retires the old CA: each certificate is followed by the new CA itself, the old CA leaves every trust file, the services restart one at a time, and the old CA with its key is archived as retired-<date>-<id>/ in the CA directory (never deleted: move it offline or remove it yourself). renew --retire-ca storage does it before the end, with a warning: clients that have not connected since the rotation started then see "CA changed" with both fingerprints and must trust the new CA again (CA rotation and handover). Credential files ctl does not manage (people's machines, CI) carry the auth CA too: after an auth CA rotation retires, issue them new ones.

With the CA key kept away from the cluster, name its directory: renew --rotate-ca storage --ca-dir DIR creates the new CA in DIR/next/, signs there, and after the run makes it current in DIR too (the old one in DIR/previous/, archived when the rotation retires with --ca-dir DIR).

Auth's two internal client CAs (credential and access, 20 years; only Scribe's services trust them, clients never see them) rotate with renew --rotate-ca credential|access: the cluster drains for a moment while auth gains a successor, which every service trusts from its start and which signs 15 minutes later. Nobody has to do anything meanwhile: people who sign in get a session from the new CA. A plain renew (the scheduled one) retires the old CA once its successor has signed for the overlap: a day for the access CA, --overlap-days (90, a session's default lifetime) for the credential CA. It renews the credentials the cluster holds first, so the new CA signs them, then retires the old CA and revokes the sessions and credentials people still hold from it: they sign in again. renew --retire-ca credential|access does the same now. Only an administrator credential is never revoked this way: if every one comes from the old CA, the retirement is refused until one is renewed. doctor shows both CAs, their keys and the day the old one retires.

Remote operation

scribe-ctl operates a cluster on another machine over the system ssh: it makes sure the remote has a scribe-ctl of this release, pushes what the command needs, and runs the same command there, passing its output and exit code through. Discovery, planning, locks, journals and execution all happen on the remote; the definition file here is the source of truth, and the secrets stay there. It replaces hand-written deploy scripts that copy a checkout to a VM, rebuild, migrate, renew credentials and open a shell.

Name the machine in the definition (or pass --remote to any command):

[clusters.prod.remote]
host = "scribe-host"      # a host name, an address, or an alias from ~/.ssh/config
user = "deploy"           # optional
port = 22                 # optional
directory = "/srv/scribe" # holds the cluster there; absolute, or relative to the login's home

Only keys, the agent and ~/.ssh/config are used: ssh runs in batch mode and never asks for a password or to accept a host key. Make ssh scribe-host true work without a prompt first (ssh-copy-id, connect once to accept the host key, or a Host alias with User, Port and IdentityFile). Private details such as host names belong in your own definition file, never in shared ones.

A new cluster there:

# A native cluster: executables built for the remote's OS and architecture,
# or none when this machine matches it (this scribe-ctl's directory is pushed).
scribe-ctl --config ./prod.toml --cluster prod --remote deploy@scribe-host:/srv/scribe setup --variant gateway --binaries ./linux-x86_64
# A docker cluster: the checkout is pushed, scribe-ctl and the images are built there.
scribe-ctl --config ./prod.toml --cluster prod --remote deploy@scribe-host:/srv/scribe setup --variant gateway --runtime docker --web --source .
scribe-ctl --config ./prod.toml up

Setup writes the definition there and brings it back here with its [remote] table, plus the summary as CLUSTER-INFO-prod.txt; after that every command (up, status, logs, df, repo list, access, renew, upgrade, add, down, where) goes there without --remote. Setup's layouts listen on the remote's loopback. For clients on other machines, write the definition here first (from scribe-ctl example), with the listen addresses, the names and addresses clients use in each [tls] table's dns and ip, and the [remote] table; setup pushes it and sets it up as written. Edits here are pushed before every command (the remote keeps its previous copy as .bak).

Upgrades push the new release: upgrade --binaries DIR (built for the remote; a newer scribe-ctl in it takes the upgrade over there) or, under docker, upgrade --source DIR (a checkout of this scribe-ctl's release, whose images are built there). scribe-ctl ssh opens a shell in the cluster's directory on the remote, and scribe-ctl ssh -- COMMAND… runs one there.

What is pushed, and where it goes in the remote directory:

A pushed checkout carries only what the image builds read, chosen without git (a checkout need not be a git one): the Rust workspace (Cargo.toml, Cargo.lock, rust-toolchain.toml, rustfmt.toml, LICENSE and every directory [workspace] members names), docker/, and apps/web whenever the checkout has it (whether or not the cluster runs the web interface; without its build outputs it is a few megabytes). Inside those, the build context's .dockerignore decides (the root one for the workspace and docker/, apps/web/.dockerignore for the web interface), read as Docker reads it for the forms Scribe's files use (# comments, ! exceptions, *, ? and ** as a whole segment); another form is refused rather than guessed at. Never pushed, wherever they are: version control, build outputs and dependencies (.git, target, target-*, node_modules, .next, *.tsbuildinfo), local state (/.scribe/_state, apps/web/deploy/local, .DS_Store) and secrets (.env, .env.local, *.credential, *.pepper, *.revision, /secrets, /pki, /cluster-state). For this repository that is about 13 MB instead of the whole working tree. rsync 3 transfers when both machines have it; otherwise a tar stream through ssh replaces the previous upload. Only the summary (CLUSTER-INFO-<cluster>.txt) and, after setup and add, the definition (and a generated gateway file) come back; CA keys, the pepper, credentials and data never leave the remote.

Each command makes one ssh connection and runs everything else through it (a ControlMaster socket in a private directory under /tmp, ended with the command; Windows' OpenSSH has none, so there each step connects on its own). A first connection the remote times out or refuses is tried four more times, after 10, 30, 60 and 90 s (a firewall that throttles bursts of new connections lets one through within that). With --source, the scribe-ctl that runs the command there is built from exactly the pushed checkout and named by a digest of it, so a changed checkout never runs a stale scribe-ctl there.

Without --source, a command runs this machine's own build there, installing it first when the remote has the same OS and architecture (or from pushed --binaries of this release). From a machine of another platform (a macOS laptop and a Linux server, say) that cannot be done, so the command runs the newest scribe-ctl of this version installed there (bin/controller), else the one bin/current names, and says so. That one knows only the commands of the checkout it was built from. bootstrap refreshes it without touching the cluster:

scribe-ctl --config ./prod.toml bootstrap --source . --plan
scribe-ctl --config ./prod.toml bootstrap --source .

It pushes the checkout to its own upload/controller/source, builds only scribe-ctl there in Docker (no images), installs it as bin/<build>/ and names it in bin/controller. bin/current, the definition there, the cluster's binary set and images, its services, lock and journal stay as they are, so it needs no restart and is safe while another command (an upgrade, say) runs there. bootstrap --binaries DIR installs a release directory built for the remote instead. A later --source setup or upgrade still names its own build in both files. doctor says which of these fixes a remote without a usable scribe-ctl (bootstrap).

A remote command does not depend on its connection: when the connection drops (or the terminal here closes), the scribe-ctl there finishes the command, or rolls it back, on its own. Run status to see how it went; a command it left pending (a stopped rollback, say) resumes when you run it again. Ctrl-C here asks the remote scribe-ctl to stop at its next safe point, as Ctrl-C does locally: through the remote terminal, or, without one (--json, redirected output), by an interrupt sent through the connection. A second Ctrl-C then leaves it running there and returns.

The remote needs a POSIX shell, tar, and for a docker cluster Docker with Compose and BuildKit. A scribe-ctl built there by Docker links the build image's glibc (Debian bookworm) and needs that or newer on the host. The reference has the full sequence and real output.

The definition file

The complete template is example.toml; scribe-ctl example prints it. All relative paths resolve against the definition's directory. [clusters.NAME.binaries] (keys auth, storage, and gateway) names where setup takes the executables it records as the first binary set; without it they are found beside scribe-ctl. Once a set is recorded, services run from <state_dir>/bin/<build>/ and [binaries] is not read again; upgrade changes the set.

ctl owns the file's layout: setup writes it in a canonical form with a comment on every setting. When setup adds a cluster to a file that defines others, or add auth or add gateway changes a cluster, ctl rewrites the file in that form and keeps the previous one as <file>.bak; comments of your own are only in the backup. variant records what setup built; a definition whose services do not match its variant is refused. Omit variant for a topology of your own (for example, plaintext and authenticated storage side by side). The definition names where secrets live but never holds one. Unknown fields are rejected to catch mistakes.

To bring services you started by hand under the controller, stop them and write a definition that names their existing files: data roots, the pepper and revision witness, the administrator credential, each store's credential, certificates, and the gateway's configuration (keep auth.ca when auth's certificate comes from the storage CA), with generate_missing = false so nothing missing is replaced. For the bare-metal guide's deployment after stage 4, in /srv/scribe/scribe-cluster.toml:

[clusters.local]
variant = "gateway"
daemon = true

[clusters.local.certificates]
generate_missing = false
ca_dir = "pki/ca"
auth_ca_dir = "pki/auth-ca"

[clusters.local.auth]
listen = "127.0.0.1:7443"
data_dir = "data/auth"
pepper_file = "secrets/auth.pepper"
revision_file = "secrets/auth.revision"
admin_credential = "secrets/admin.credential"
[clusters.local.auth.tls]
cert = "pki/auth/server.pem"
key = "pki/auth/server-key.pem"

[[clusters.local.storage]]
name = "storage-a"
listen = "127.0.0.1:7447"
data_dir = "data/storage-a"
auth = true
namespace = "studio"
credential = "secrets/storage-a.credential"
[clusters.local.storage.tls]
cert = "pki/storage-a/server.pem"
key = "pki/storage-a/server-key.pem"

[[clusters.local.storage]]
name = "storage-b"
listen = "127.0.0.1:7448"
data_dir = "data/storage-b"
auth = true
namespace = "studio"
credential = "secrets/storage-b.credential"
[clusters.local.storage.tls]
cert = "pki/storage-b/server.pem"
key = "pki/storage-b/server-key.pem"

[clusters.local.gateway]
config = "scribe-gateway.toml"

scribe-ctl starts each store from its table alone and never reads storage-a.toml or storage-b.toml: a setting you changed there beyond the listen address, data root and [auth] table does not carry over. The gateway keeps reading scribe-gateway.toml. Run and log directories move to state_dir (cluster-state/local). setup --plan then shows that it only confirms identities with auth; setup records the executables and writes CLUSTER-INFO.txt, and up starts them. The CLI reference explains binary/configuration lookup and command-line selection.

Named clusters and selection

Every definition lives under clusters.NAME. Names use ASCII letters, digits, hyphens, and underscores. For example:

[clusters.local]
daemon = true
[[clusters.local.storage]]
name = "assets"
listen = "127.0.0.1:7447"
data_dir = "data/local/assets"
auth = false

[clusters.staging]
daemon = true
[[clusters.staging.storage]]
name = "assets"
listen = "127.0.0.1:8447"
data_dir = "data/staging/assets"
auth = false

--cluster NAME is global, so it can precede or follow the command. With one cluster it is optional. With multiple clusters, setup, up, down, df, repo list, and logs require it; an ambiguous command exits before changing anything. status and where show every cluster in name order, or only the selected one:

scribe-ctl --cluster local setup
scribe-ctl up --cluster local
scribe-ctl status
scribe-ctl status --cluster staging
scribe-ctl logs assets --cluster local
scribe-ctl down --cluster local

setup --cluster staging --variant storage adds a cluster to an existing file: its paths start with staging/, and its ports move up by a multiple of 100 so both can run at once (or give --port-offset).

Clusters start and stop independently and keep separate logs. Default state_dir is cluster-state/NAME; default certificates.ca_dir is pki/NAME/ca and default certificates.auth_ca_dir is pki/NAME/auth-ca. Other relative paths remain relative to the TOML file, so provide separate data, PEM, account and credential paths as needed. Overlapping state or data roots are rejected across clusters, including paths through symlinks. Shared listen ports are allowed for alternative clusters, but those clusters cannot run on the same address simultaneously.

Accounts and pepper

The separate accounts_file contains password-file references, never inline passwords. Password paths resolve against the accounts file's directory:

[[users]]
name = "alice"
password_file = "passwords/alice"
administrator = false

[[users]]
name = "operator"
password_file = "passwords/operator"
administrator = true

Use private password files containing 12–1024 UTF-8 bytes. A trailing line ending is allowed. Keep this file and secrets out of version control. Omit accounts_file if no account bootstrap is wanted.

The auth table names the files setup creates and every later start reads:

[clusters.local.auth]
listen = "127.0.0.1:7443"
data_dir = "data/auth"
pepper_file = "secrets/auth.pepper"
# Revision witness: keep it (like the pepper) outside the data directory and its backups.
revision_file = "secrets/auth.revision"
# The bootstrap administrator's credential, written by setup. Keep it private.
admin_credential = "secrets/admin.credential"
[clusters.local.auth.tls]
cert = "pki/auth/server.pem"
key = "pki/auth/server-key.pem"

Setup creates a missing auth database by running scribe-auth init: it writes the required pepper to auth.pepper_file, the revision witness to auth.revision_file, and the bootstrap administrator's credential file to auth.admin_credential. The credential names auth's listen address (a wildcard bind becomes loopback) and the CA of auth's certificate. Every auth start receives the pepper and the witness. Existing auth databases require their original pepper and witness: the controller never regenerates either for an existing database. Retain and back up the pepper separately from the database, and keep the witness out of the database's backups, as described in auth.md.

Setup also registers authenticated storage identities and provisions their server credentials (and the gateway's and inventory credentials), then bootstraps users:

• A missing credential file is issued for 365 days.
• An existing file must hold a live credential of that holder; a file of another subject, or a revoked one, is refused.
• An existing credential is kept, also within 30 days of its expiry: renew replaces it (Renew service credentials). scribe-ctl status and where warn 30 days ahead.

Repeat runs skip existing users without reading their password files or changing their password, administrator status, or enabled state. The initial admin account is consequently left unchanged too. To add accounts later, add them to accounts_file and run setup again; it works while the cluster runs:

scribe-ctl --config ./scribe-cluster.toml setup

When the cluster is stopped, setup starts auth temporarily and stops it afterward. An interrupted bootstrap resumes on the next run; both of its operation ids are durable before either request. Keep state_dir and the original account/password files until the operation completes. Removing a user from the file does not delete their account; use auth administration commands to disable an account or change its password.

The administrator credential is required for provisioning and account bootstrap; ordinary up needs the pepper, the revision witness, and the server credentials. Live inventory through scribe-ctl needs a separate monitor_credential: the credential of a service account with a namespace reader grant in the store's namespace, which setup provisions; central administrator status alone reads nothing (see live storage inventory).

Certificates and topology

The controller uses two server-certificate CAs:

[clusters.local.certificates]
generate_missing = true
# Storage and gateway certificates; auth's comes from its own CA, so no storage
# key can ever pass as auth.
ca_dir = "pki/ca"
auth_ca_dir = "pki/auth-ca"

ca_dir signs storage and gateway certificates; auth_ca_dir signs only auth's certificate. Credential files name the auth CA as the only trust for auth, so a stolen storage or gateway key cannot impersonate auth to their holders. The two directories must differ. Both are server-certificate CAs only; client certificates come from auth's own CAs (see identity: device keys and certificates).

certificates.generate_missing = true permits setup to create a missing CA (valid 3650 days) and missing leaf certificate/key pairs (365 days). Existing PEMs are preserved and leaf pairs are validated. A partial pair is refused: preserve/move the surviving file before explicitly regenerating the pair (or setup moves its own interrupted pair aside; see rerun, resume, and what setup refuses). This also applies to incomplete CA material. Certificates are never silently rotated on up. status warns when a server certificate is within 30 days of expiry; renew reissues it for the key it has, while the cluster runs.

While generate_missing is in use, both CA private keys (ca-key.pem in each directory) sit on the controller host, online. Once the certificates you need exist, move the CA keys to offline storage and set generate_missing = false; bring a key back only to issue a certificate. Setup names both keys in CLUSTER-INFO.txt with that advice.

Default generated SANs cover localhost, IPv4/IPv6 loopback, and the concrete listen IP. Set dns and/or ip arrays in a service's tls table for other names; explicit arrays replace these defaults. auth.ca can select an existing trust bundle for auth's certificate, for example when adopting an auth service whose certificate another CA issued; by default the controller uses certificates.auth_ca_dir/ca.pem. The administrator credential that setup creates names this CA, and credentials issued later copy it from there.

Auth requires TLS and a fixed, nonzero listen port so bootstrap retries retain the same endpoint. Authenticated storage requires auth = true, namespace, credential, and a tls table:

[[clusters.local.storage]]
name = "assets"
listen = "127.0.0.1:7447"
data_dir = "data/assets"
auth = true
namespace = "studio"
# This server's own credential, issued by setup. Keep it private.
credential = "secrets/assets.credential"
[clusters.local.storage.tls]
cert = "pki/assets/server.pem"
key = "pki/assets/server-key.pem"

In the current storage adapter TLS is coupled to auth integration. Standalone storage uses auth = false and no TLS/auth fields; non-loopback plaintext requires allow_non_loopback = true explicitly. Storage port zero is allowed, with its assigned endpoint shown by status.

Each data root and the state directory must be distinct and nonoverlapping. Private keys, CA directories, the pepper, the revision witness, the administrator credential, and service credentials must be outside every service data root. Unknown TOML fields are rejected to catch configuration mistakes.

The certificates and credentials themselves, with their lifetimes and how to replace each one with or without scribe-ctl, are in certificates and credentials.

Lifecycle and diagnostics

setup is explicit provisioning; up never initializes data or creates secrets. daemon = true (which setup writes) makes up detach and return once all services report readiness. Without it, the default is foreground. up --daemon and up --foreground override TOML. In foreground mode, Ctrl-C requests graceful shutdown.

The supervisor starts auth first, then storage, then gateway. Shutdown drains gateway before storage, and storage before auth. Startup failures and unexpected service exits stop the remaining services. There is no automatic restart policy. Service output is appended to state_dir/logs/<service>.log; detached supervisor output is in supervisor.log. logs prints a bounded tail, not a live stream. These files are not automatically rotated.

status shows an aligned service table and warns about credentials and server certificates within 30 days of expiry, read from the files themselves; --json selects machine-readable output for every public controller command. Status is read-only.

doctor is the full health check, read-only too: the definition and the files it names, the binary set against this scribe-ctl, free space against the upgrade's backup reserve, each data root's formats against the active set, auth's data-root permissions and secret files, every expiry, the services, their ports and the gateway's inventory, an interrupted command, kept backups, and under docker the images, the build cache and what an adoption left behind. Each check is ok, info, warn or fail, with the fix beside it; the exit status is 0, 1 (warnings only) or 3 (a failure), so a scheduler can run it and alert on nonzero. With a remote cluster it checks ssh from here and runs the remote's own doctor there, as one report, without pushing or installing anything. See doctor.

scribe-ctl --config ./scribe-cluster.toml doctor
scribe-ctl --config ./scribe-cluster.toml doctor --json   # for a monitor: key on each check's id

If the controller exits unexpectedly, status reports unmanaged services. Run down to stop them before restarting the cluster.

startup_timeout_secs defaults to 30 (allowed range 1–300), applies per service on startup, and bounds how long down waits. A shutdown timeout reports that services are still draining; it never force-kills a durable commit. Leave the supervisor running and retry status/down.

Controller diagnostics and build-time timing support are documented in logging; option syntax is in the CLI reference.

Optional managed gateway

Point the cluster at the same configuration used by the standalone gateway:

[clusters.local.gateway]
config = "scribe-gateway.toml"
require_healthy = true

The gateway file follows gateway configuration. Its paths resolve against its own directory. Use fixed storage listen ports so its instance endpoints remain valid across restarts. Gateway-only clusters can supervise a gateway connected to storage managed elsewhere.

Setup initializes the gateway data root and includes its TLS certificate/key pair in certificate provisioning, with the default SANs described above. Existing roots and PEM pairs are preserved. setup --variant gateway and add gateway also write the gateway configuration, scribe-gateway.toml, beside the definition.

For a TLS gateway, setup also provisions the gateway's identity:

• It creates the scribe-gateway service account if it is missing.
• For every namespace the gateway's instances serve, it sets the account's namespace reader grant (inventory) and delegate grant (acting for users; see gateways and delegation). Setting an existing grant changes nothing.
• It issues a 365-day service credential to the gateway configuration's [tls] credential path when the file is missing; renew replaces it before it expires. A file that does not hold a live credential of scribe-gateway is refused.

A gateway instance that matches a managed store's endpoint must name that store's namespace; the controller refuses a mismatch.

By default, startup waits for a listening gateway and fresh, successful inventory from every configured instance. An unhealthy instance causes startup to time out and the cluster to drain. Set require_healthy = false to allow startup with unavailable instances. Later backend outages are handled by the gateway and do not automatically stop the cluster. scribe-ctl status shows process lifecycle; scribe-gateway --config scribe-gateway.toml status shows backend inventory.

Output for people and scripts

Lifecycle commands report progress on stderr and a service table on stdout. Use --json for machine-readable results. Foreground up emits its final JSON only after shutdown; daemon mode returns after readiness. Status describes process lifecycle, whereas the gateway's status describes backend observations.

The JSON output guide describes the result fields and error schema. Logging covers terminal colors and diagnostics. Service and supervisor logs are separate from command results and are not automatically rotated by the controller.

Live storage inventory

Query all configured stores, including additional backends in the gateway config:

scribe-ctl --cluster local status --storage
scribe-ctl --cluster local df
scribe-ctl --cluster local repo list --json

status --storage combines lifecycle state, disk usage and repositories; without --cluster it covers all named clusters. df summarizes storage sizes and repo list includes each repository's server, ID, head and apparent size. Servers are queried directly while running. Errors are retained beside successful results and produce exit 1. Free disk space is per server because roots may share a filesystem. These are live samples, not an atomic cluster snapshot.

Terminal inventory uses aligned tables and readable IEC sizes (KiB, MiB, GiB, and larger units). Use --json for automation: size fields remain exact integer byte counts, with no rounding or unit suffixes.

For authenticated stores, add these optional fields inside the existing [[clusters.local.storage]] table, before its [clusters.local.storage.tls] table:

monitor_credential = "secrets/monitor.credential"
ca = "pki/ca/ca.pem"

The monitor credential must read the store's namespace: the credential of a service account with a namespace reader grant there. Central auth administrator status alone does not grant that access, and the store's own server credential is not a user credential. Setup provisions it: the scribe-monitor service account, a namespace reader grant in every namespace whose store names the file, and the credential itself, which renew replaces like the others. A file that holds another account's credential is refused. The auth and gateway variants configure secrets/monitor.credential for their store.

Inventory does not change permissions. Keep the credential file outside service data roots. The ca field defaults to the CA of an exact endpoint match in the gateway configuration, then to certificates.ca_dir/ca.pem. Only the store's own monitor_credential supplies a credential: an authenticated backend listed only in the gateway configuration is reported as a per-server error. Plaintext local stores need neither field.

Take over an existing Docker deployment

A deployment started before scribe-ctl ran it (with the removed docker/init.sh, in basic, auth or gateway mode, optionally with the web interface's own Compose project beside it, or the same shapes started by hand with Compose, such as the Docker Compose guide's) comes under scribe-ctl with adopt, without moving or re-creating its data or secrets.

For the Docker Compose guide's project in /srv/scribe (project scribe), with the web interface's project (scribe-web) beside it, from a checkout of this release:

scribe-ctl --config ~/scribe/scribe-cluster.toml adopt --runtime docker \
  --admin-credential /srv/scribe/secrets/admin/admin.credential --source . --plan
scribe-ctl --config ~/scribe/scribe-cluster.toml adopt --runtime docker \
  --admin-credential /srv/scribe/secrets/admin/admin.credential --source .

Put the definition in a directory of its own, owned by the user the containers run as (SCRIBE_UID/SCRIBE_GID in the project's .env): scribe-ctl's containers run as that directory's owner. adopt finds the Compose projects (or takes them from --project scribe --project scribe-web), reads from their containers where every data root, secret, certificate, credential and configuration is on the host, and writes a definition that points at them where they are. The administrator credential is not in any serving container, so it is named with --admin-credential (docker/init.sh wrote it as operator/admin.credential in its state directory). --source builds this release's images, as setup does; the web interface always needs it. --plan runs every check and shows the definition it would write; anything it cannot carry over (another container in the project, a flag or setting scribe-ctl's containers would not pass, a data root in a named volume, a credential that names auth differently, auth before the identity v2 switch, secrets others can read) is refused by name.

Adopting stops the old projects (docker compose -p NAME stop: nothing is removed), copies the web database once out of its named volume into data/web beside the definition (the volume is kept), starts the cluster's own project on the same ports and checks that it serves. If that fails, the old projects start again exactly as they ran and the cluster is taken out of the definition again. Clients keep their endpoints, sessions and trust.

Afterwards the cluster is managed like any docker cluster (status, upgrade --source, renew, where). What is left of the old deployment is yours to remove once the cluster has proven itself; the completed run lists it:

docker compose -p scribe down              # containers and networks; never -v
docker compose -p scribe-web down
docker volume rm scribe-web_database       # the web database's old volume, once you no longer need it

(docker/init.sh's projects were scribe-gateway, scribe-auth or scribe-basic; the completed run names the ones it stopped.)

Keep the old Compose files and checkout until then: the old projects' files are what a later docker compose -p NAME up would need, and the web interface's other volumes (objects, search) stay in use by the adopted cluster. The reference lists every check and shows a real plan and rollback.