Previous: gateway scaling · All tutorials · Next: web interface
The cluster from stages 1–3 runs auth, two stores and a gateway. This stage
walks through the everyday operations: what runs, a full health check, where
every file is, logs, storage inventory, who has access, renewing credentials and
certificates and rotating a CA, an upgrade to a new build, and running a
cluster on another machine. Work from scribe-demo; start the cluster first if it is stopped
(scribe-ctl --config scribe-cluster.toml up).
1. What runs, and where everything is
scribe-ctl --config scribe-cluster.toml statusScribe · local · running
SERVICE STATE PID ENDPOINT
auth running 17999 tls://127.0.0.1:7443
storage-a running 18000 tls://127.0.0.1:7447
storage-b running 18001 tls://127.0.0.1:7448
gateway running 18002 tls://127.0.0.1:7557
Logs /home/you/scribe-demo/cluster-state/local/logs
Binaries 0.1.0 (build 86da462a7d46.dirty-20260927T124201) · /home/you/scribe-demo/cluster-state/local/bin/86da462a7d46.dirty-20260927T124201status is read-only. It also warns 30 days before a credential or server
certificate expires. where prints the cluster summary, the same text as
CLUSTER-INFO.txt, grouped by what to do with each file:
scribe-ctl --config scribe-cluster.toml whereScribe cluster local · gateway variant
Definition: /home/you/scribe-demo/scribe-cluster.toml
Files below are relative to its directory.
Endpoints
auth tls://127.0.0.1:7443
storage-a tls://127.0.0.1:7447
storage-b tls://127.0.0.1:7448
gateway tls://127.0.0.1:7557 (clients connect here)
Trust: public: give these to every client and to the web interface
pki/ca/ca.pem
storage and gateway CA certificate; clients verify servers with it (--tls-ca)
pki/auth-ca/ca.pem
auth CA certificate; clients verify auth with it at login (--auth-ca)
CA private keys: back these up offline and never share them; setup needs them only to issue server certificates
pki/ca/ca-key.pem
signs storage and gateway certificates
pki/auth-ca/ca-key.pem
signs auth's certificate
Secrets: private to this machine: never share them or put them in version control
secrets/auth.pepper
auth's password pepper; back it up apart from the auth data, which cannot check passwords without it
secrets/admin.credential
the bootstrap administrator's credential (scribe-auth --admin-credential)
...
Auth revision witness: keep it outside every auth backup; auth refuses to start behind it
secrets/auth.revision
auth's latest security revision
...
Data: back these up (docs/migration-backup-restore.md); never run two copies at once
data/auth
auth: accounts, credentials, grants
data/storage-a
storage storage-a: repositories
data/storage-b
storage storage-b: repositories
data/gateway
gateway: placement ledger
...The sections are the backup plan: data roots together (see backups), the pepper apart from them, the witness never inside an auth backup, the CA private keys offline, and only the trust files given to clients. The summary ends with the next commands, such as adding a person.
Check everything with doctor
status answers what runs; doctor checks everything the cluster depends on,
read-only: the definition and every file it names, the binary set, free space
for an upgrade's backups, each data root's formats against the executables,
permissions of the secrets, every expiry, the services and their ports, the
gateway's inventory, and an interrupted command:
scribe-ctl --config scribe-cluster.toml doctorScribe · local · doctor · native runtime
Definition /home/you/scribe-demo/scribe-cluster.toml
Read only: nothing is written, locked, started or stopped.
...
DATA
ok data.auth data/auth: auth_schema 4, as the active scribe-auth reads and writes
ok data.storage-a data/storage-a: catalog_schema 1, envelope 1, object_store 1, repository_schema 3, as the active scribe-server reads and writes
ok data.gateway data/gateway: route_store_schema 2, as the active scribe-gateway reads and writes
SECRETS AND PERMISSIONS
ok permissions.auth-root data/auth is reachable by its owner only
ok permissions.secrets 10 secret files are private to their owner
ok auth.witness secrets/auth.revision, outside auth's data root
ok auth.pepper secrets/auth.pepper, outside auth's data root
CERTIFICATES AND CREDENTIALS
ok expiry.administrator-credential administrator credential (secrets/admin.credential): does not expire
ok expiry.auth-certificate auth certificate (pki/auth/server.pem): expires in 364 days (2027-09-27)
...
ok expiry.storage-and-gateway-ca storage and gateway CA (pki/ca/ca.pem): expires in 3649 days (2036-09-24)
ok expiry.auth-ca auth CA (pki/auth-ca/ca.pem): expires in 3649 days (2036-09-24)
ok ca.key.storage the storage and gateway CA has an ML-DSA-65 key
ok ca.key.auth the auth CA has an ML-DSA-65 key
ok ca.internal.credential auth's internal credential CA is valid until 2046-09-27
ok ca.internal.access auth's internal access CA is valid until 2046-09-27
...Every check is ok, info, warn or fail, with the fix beside it. The exit status
is 0 when nothing warns, 1 with warnings only and 3 with a failure, so a
scheduler can run it and alert on nonzero (--json keys each check by its
id). See lifecycle and diagnostics.
2. Logs
scribe-ctl --config scribe-cluster.toml logs gateway --lines 5
scribe-ctl --config scribe-cluster.toml logs storage-a
scribe-ctl --config scribe-cluster.toml logs supervisorScribe · local · gateway logs
{"timestamp":"2026-09-27T12:44:33.677Z","level":"INFO","message":"access certificate issuers loaded from auth","event":"auth.trust_loaded","access_cas":1,"target":"scribe_auth"}
gateway listening on tls://127.0.0.1:7557
{"timestamp":"2026-09-27T12:44:33.677Z","level":"INFO","message":"listening","event":"server.listening","addr":"127.0.0.1:7557","storage_workers":36,"query_workers":9,"transfer_workers":18,"handshake_workers":8,"target":"scribe_protocol_tcp::server"}Each service appends to cluster-state/local/logs/<service>.log; logs shows a
bounded tail, not a live stream. supervisor is the detached controller's own
log (No log lines yet. while nothing went wrong). These files are not rotated
automatically. The cluster guide
explains startup failures, shutdown timeouts, and unmanaged processes.
3. Storage inventory
scribe-ctl --config scribe-cluster.toml df
scribe-ctl --config scribe-cluster.toml status --storageScribe · local · storage inventory
SERVER REPOS USED AVAILABLE ENDPOINT
storage-a 2 2.80 MiB 436.11 GiB tls://127.0.0.1:7447
storage-b 1 2.25 MiB 436.11 GiB tls://127.0.0.1:7448
Total: 3 repositories · 5.05 MiB useddf asks every store directly, as the scribe-monitor service account whose
credential add auth issued; repo list (stage 3) adds each repository, and
status --storage combines both with the service table. These are live
samples; --json gives exact byte counts for scripts.
4. Who can do what
scribe-ctl --config scribe-cluster.toml access --user bobScribe · local · access
ACCOUNT KIND STATE SCOPE ACCESS DETAIL
bob person enabled repository studio/689a965e479c1decf32f3c9ca7e3f913 writer
bob person enabled sessions 2 signed in latest expires 2026-12-26Bob has two sessions: one for storage A from stage 2 and one for the gateway
from stage 3. Without --user, access lists every account (stage 2 showed
it). It reads through the administrator credential and needs auth running. To
change access, use scribe-auth as in stage 2.
To create accounts from a file instead, see
accounts and pepper.
5. Renew credentials and certificates
Every service credential (storage-a, storage-b, the gateway's,
scribe-monitor's) and every service's TLS certificate is valid for 365 days.
renew replaces each one that expires within 30 days, and does nothing
otherwise:
scribe-ctl --config scribe-cluster.toml renew --plan
scribe-ctl --config scribe-cluster.toml renewScribe · local · renew plan · gateway variant · native runtime
...
CHECKS
ok credentials nothing to renew: 5 credentials are valid for 30 more days; the first to expire is the storage-a server credential on 2027-09-27
ok certificates 4 server certificates are valid for 30 more days; the first to expire is auth's on 2027-09-27
ok cluster running; no service needs a restart
...
Scribe · local · Unchanged: nothing to renew: 5 credentials are valid for 30 more days; the first to expire is the storage-a server credential on 2027-09-27
4 server certificates are valid for 30 more days; the first to expire is auth's on 2027-09-27When something is due, renew issues the replacement, puts it in place,
restarts the services that read it, proves it works, and only then revokes the
old one; if a proof fails, the old credentials go back and it exits 1. Run it
daily from a scheduler: renew service
credentials has cron, systemd
and Task Scheduler examples.
- Server certificates are reissued for the key they already have, by the
CA that signed them, so clients notice nothing. That needs the CA's private
key:
ca-key.pembeside the cluster, or, once you have moved the keys offline aswhereadvises,renew --ca-dir DIR(storage and gateway CA) and--auth-ca-dir DIR(the auth CA) pointing at them. - The administrator credential setup wrote never expires. Once the
definition names an expiring one (see administrator
credentials),
renewreplaces it too. - The web interface's own certificate (stage 5) stays yours.
Rotate a CA
The two server-certificate CAs are valid for 10 years; status, doctor and
renew say so a year ahead. A rotation replaces one without asking any client
to trust something by hand, as long as the client connects during an overlap.
It is never started by a scheduled renew:
scribe-ctl --config scribe-cluster.toml renew --rotate-ca storage --plan
scribe-ctl --config scribe-cluster.toml renew --rotate-ca storage ...
ROTATION
The new storage and gateway CA is current: BLAKE3:68bf8bcb6e2c0679ca33473f8dd705b5c83f12c9ff82a0ce21bcfa61f0904991
The old one (BLAKE3:26e411692a534f4d68414ab4226585414fa41a489382371fd9bc8be07f90bd7b) stays trusted through the cross certificate for 90 days; a plain renew then retires it.
Clients learn the new CA when they connect in that time; give new clients the new ca.pem.Both stores and the gateway now present their certificate followed by a
handover certificate the old CA signed for the new one. A client that trusts
only the old CA verifies that chain and saves the new CA by itself (scribe trust show then lists both); the next scheduled renew after the overlap
retires the old CA, and clients drop it. status shows the day:
CA the storage and gateway CA is being rotated since 2026-09-27; a plain renew retires the old CA on 2026-12-26 (`scribe-ctl --config /home/you/scribe-demo/scribe-cluster.toml renew --retire-ca storage` does it now)--rotate-ca auth rotates the auth CA the same way, and --rotate-ca credential|access auth's two internal CAs. What happens to clients is in
when a server's CA changes;
every CA and how to replace each without scribe-ctl is in certificates and
credentials.
6. Upgrade to a new build
After pulling new source into your checkout, one line from the checkout builds
it, upgrades the cluster, and installs the new executables only if the
upgrade succeeded. Run it from the checkout, with the path of your
scribe-demo directory. Look at the plan first (-- --plan passes --plan
to scribe-ctl upgrade; nothing is installed):
./scripts/install.sh --upgrade --config ~/scribe-demo/scribe-cluster.toml -- --plan
./scripts/install.sh --upgrade --config ~/scribe-demo/scribe-cluster.tomlScribe · building release binaries (build tutorial-next)
...
Scribe · local · upgrade plan · gateway variant · native runtime
...
CHECKS
ok release 0.1.0 (build 86da462a7d46.dirty-20260927T124201) → 0.1.0 (build tutorial-next): the same version, another build
ok installed set 0.1.0 (build 86da462a7d46.dirty-20260927T124201) in cluster-state/local/bin/86da462a7d46.dirty-20260927T124201
ok new set /home/you/scribe/target/install/aarch64-apple-darwin/release: scribe-ctl, scribe-auth, scribe-server, scribe-gateway 0.1.0 (build tutorial-next) for macos/aarch64, and the scribe client (not installed)
ok controller scribe-ctl 0.1.0 (build tutorial-next)
ok formats unchanged: every executable reads and writes what the active set does
ok cluster running; drained after the new set is installed
ok auth data auth_schema 4; no migration: its formats do not change (no dry run: auth serves it and holds its data-root lock)
ok storage-a data catalog_schema 1, envelope 1, object_store 1, repository_schema 3; no migration: its formats do not change (no dry run: storage-a serves it and holds its data-root lock)
ok storage-b data catalog_schema 1, envelope 1, object_store 1, repository_schema 3; no migration: its formats do not change (no dry run: storage-b serves it and holds its data-root lock)
ok gateway data route_store_schema 2; no migration
STEPS
1. Install scribe-ctl, scribe-auth, scribe-server, scribe-gateway 0.1.0 (build tutorial-next) side by side in cluster-state/local/bin/tutorial-next (from /home/you/scribe/target/install/aarch64-apple-darwin/release)
side by side with the active set, before anything stops
2. Drain the running cluster
roots are backed up, migrated and switched only while nothing holds them
3. Switch the active set from 86da462a7d46.dirty-20260927T124201 to tutorial-next (bin/current)
services start from the set bin/current names
4. Start the cluster with tutorial-next and wait until every service is ready
the health check: every service must become ready
5. Write the cluster summary CLUSTER-INFO.txt
the summary names the active set
ROLLBACK (if step 3 or a later one fails)
6. Drain whatever of the cluster runs
...
Plan only: the installed executables are unchanged.The build id names each build (the checkout's commit, with .dirty-<time> for
local changes, or SCRIBE_BUILD_ID when you set it; this run set
SCRIBE_BUILD_ID=tutorial-next). The real run:
Scribe · local · Upgrade complete
installed 0.1.0 (build tutorial-next) in cluster-state/local/bin/tutorial-next
drained the cluster
switched the active set from 86da462a7d46.dirty-20260927T124201 to tutorial-next
every service is ready, running tutorial-next
wrote CLUSTER-INFO.txt
...
Installed /home/you/.local/bin/scribe
Installed /home/you/.local/bin/scribe-server
...
Installation complete. No profiles changed.What scribe-ctl did for you
The new release's own scribe-ctl ran the upgrade, because only it knows that
release's migrations. It checked that the new executables are one complete
release for this machine and not older than the running one, compared the
formats each reads and writes, and ran migration dry runs where it could. It
installed the new set beside the old one while the cluster kept serving, then
drained it, switched bin/current, started the new set and waited until every
service was ready. When a format changes, it also backs up each root a
migration changes before migrating it, and if anything fails once the
cluster is drained, it rolls back by itself: old executables, restored roots,
the cluster running as before. Nothing is deleted: where lists the earlier set, kept
until you remove it. See upgrade to a new
release.
With a downloaded release instead of a checkout, unpack it into a directory of
its own and give that to scribe-ctl directly:
scribe-ctl --config scribe-cluster.toml upgrade --binaries /path/to/release --plan
scribe-ctl --config scribe-cluster.toml upgrade --binaries /path/to/releaseA newer scribe-ctl in that directory takes the upgrade over; use it (or copy
it where you keep scribe-ctl) from then on.
Check that clients still work: (cd work/alice && alice sync) and alice repo list show the same repositories.
7. Stop, or run attached
scribe-ctl --config scribe-cluster.toml down
scribe-ctl --config scribe-cluster.toml up --foregrounddown drains the gateway, then storage, then auth, and never force-kills a
durable commit. up --foreground keeps the supervisor in your terminal
(overriding daemon = true); Ctrl-C requests the same graceful shutdown.
Neither arranges startup after a reboot or restarts a failed service; see
production deployment.
8. Operate a cluster on another machine, or take one over
Everything in this stage works on a cluster on another machine too: name it
in the definition's [remote] table (or pass --remote deploy@scribe.example.com:/srv/scribe), and every command runs there over the
system ssh, with the definition here as the source of truth and the secrets
staying there. scribe-ctl ssh opens a shell in the cluster's directory. See
remote operation.
A Docker deployment that scribe-ctl did not create (Compose projects you run
by hand, for example these) comes under the
controller with adopt, without moving its data or secrets: see take over an
existing Docker deployment.
Native services you started by hand are brought under it with a definition that
names their files (the definition file).
Continue with 5. Start the web interface. For
ongoing care, see the operations guide and the
maintenance guide; for the same operations
without scribe-ctl, see certificates and credentials
and upgrades by hand.