4. Manage services with ctl

Previous: gateway scaling · All tutorials · Next: web interface

The cluster from stages 1–3 runs auth, two stores and a gateway. This stage walks through the everyday operations: what runs, a full health check, where every file is, logs, storage inventory, who has access, renewing credentials and certificates and rotating a CA, an upgrade to a new build, and running a cluster on another machine. Work from scribe-demo; start the cluster first if it is stopped (scribe-ctl --config scribe-cluster.toml up).

1. What runs, and where everything is

scribe-ctl --config scribe-cluster.toml status

text

Scribe · local · running
  SERVICE    STATE        PID  ENDPOINT
  auth       running    17999  tls://127.0.0.1:7443
  storage-a  running    18000  tls://127.0.0.1:7447
  storage-b  running    18001  tls://127.0.0.1:7448
  gateway    running    18002  tls://127.0.0.1:7557

  Logs  /home/you/scribe-demo/cluster-state/local/logs
  Binaries  0.1.0 (build 86da462a7d46.dirty-20260927T124201) · /home/you/scribe-demo/cluster-state/local/bin/86da462a7d46.dirty-20260927T124201

status is read-only. It also warns 30 days before a credential or server certificate expires. where prints the cluster summary, the same text as CLUSTER-INFO.txt, grouped by what to do with each file:

scribe-ctl --config scribe-cluster.toml where

text

Scribe cluster local · gateway variant
Definition: /home/you/scribe-demo/scribe-cluster.toml
Files below are relative to its directory.

Endpoints
  auth       tls://127.0.0.1:7443
  storage-a  tls://127.0.0.1:7447
  storage-b  tls://127.0.0.1:7448
  gateway    tls://127.0.0.1:7557  (clients connect here)

Trust: public: give these to every client and to the web interface
  pki/ca/ca.pem
      storage and gateway CA certificate; clients verify servers with it (--tls-ca)
  pki/auth-ca/ca.pem
      auth CA certificate; clients verify auth with it at login (--auth-ca)

CA private keys: back these up offline and never share them; setup needs them only to issue server certificates
  pki/ca/ca-key.pem
      signs storage and gateway certificates
  pki/auth-ca/ca-key.pem
      signs auth's certificate

Secrets: private to this machine: never share them or put them in version control
  secrets/auth.pepper
      auth's password pepper; back it up apart from the auth data, which cannot check passwords without it
  secrets/admin.credential
      the bootstrap administrator's credential (scribe-auth --admin-credential)
  ...

Auth revision witness: keep it outside every auth backup; auth refuses to start behind it
  secrets/auth.revision
      auth's latest security revision
...
Data: back these up (docs/migration-backup-restore.md); never run two copies at once
  data/auth
      auth: accounts, credentials, grants
  data/storage-a
      storage storage-a: repositories
  data/storage-b
      storage storage-b: repositories
  data/gateway
      gateway: placement ledger
...

The sections are the backup plan: data roots together (see backups), the pepper apart from them, the witness never inside an auth backup, the CA private keys offline, and only the trust files given to clients. The summary ends with the next commands, such as adding a person.

Check everything with doctor

status answers what runs; doctor checks everything the cluster depends on, read-only: the definition and every file it names, the binary set, free space for an upgrade's backups, each data root's formats against the executables, permissions of the secrets, every expiry, the services and their ports, the gateway's inventory, and an interrupted command:

scribe-ctl --config scribe-cluster.toml doctor

text

Scribe · local · doctor · native runtime
  Definition  /home/you/scribe-demo/scribe-cluster.toml
  Read only: nothing is written, locked, started or stopped.
  ...
  DATA
    ok    data.auth       data/auth: auth_schema 4, as the active scribe-auth reads and writes
    ok    data.storage-a  data/storage-a: catalog_schema 1, envelope 1, object_store 1, repository_schema 3, as the active scribe-server reads and writes
    ok    data.gateway    data/gateway: route_store_schema 2, as the active scribe-gateway reads and writes

  SECRETS AND PERMISSIONS
    ok    permissions.auth-root  data/auth is reachable by its owner only
    ok    permissions.secrets    10 secret files are private to their owner
    ok    auth.witness           secrets/auth.revision, outside auth's data root
    ok    auth.pepper            secrets/auth.pepper, outside auth's data root

  CERTIFICATES AND CREDENTIALS
    ok    expiry.administrator-credential     administrator credential (secrets/admin.credential): does not expire
    ok    expiry.auth-certificate             auth certificate (pki/auth/server.pem): expires in 364 days (2027-09-27)
    ...
    ok    expiry.storage-and-gateway-ca       storage and gateway CA (pki/ca/ca.pem): expires in 3649 days (2036-09-24)
    ok    expiry.auth-ca                      auth CA (pki/auth-ca/ca.pem): expires in 3649 days (2036-09-24)
    ok    ca.key.storage                      the storage and gateway CA has an ML-DSA-65 key
    ok    ca.key.auth                         the auth CA has an ML-DSA-65 key
    ok    ca.internal.credential              auth's internal credential CA is valid until 2046-09-27
    ok    ca.internal.access                  auth's internal access CA is valid until 2046-09-27
  ...

Every check is ok, info, warn or fail, with the fix beside it. The exit status is 0 when nothing warns, 1 with warnings only and 3 with a failure, so a scheduler can run it and alert on nonzero (--json keys each check by its id). See lifecycle and diagnostics.

2. Logs

scribe-ctl --config scribe-cluster.toml logs gateway --lines 5
scribe-ctl --config scribe-cluster.toml logs storage-a
scribe-ctl --config scribe-cluster.toml logs supervisor

text

Scribe · local · gateway logs
{"timestamp":"2026-09-27T12:44:33.677Z","level":"INFO","message":"access certificate issuers loaded from auth","event":"auth.trust_loaded","access_cas":1,"target":"scribe_auth"}
gateway listening on tls://127.0.0.1:7557
{"timestamp":"2026-09-27T12:44:33.677Z","level":"INFO","message":"listening","event":"server.listening","addr":"127.0.0.1:7557","storage_workers":36,"query_workers":9,"transfer_workers":18,"handshake_workers":8,"target":"scribe_protocol_tcp::server"}

Each service appends to cluster-state/local/logs/<service>.log; logs shows a bounded tail, not a live stream. supervisor is the detached controller's own log (No log lines yet. while nothing went wrong). These files are not rotated automatically. The cluster guide explains startup failures, shutdown timeouts, and unmanaged processes.

3. Storage inventory

scribe-ctl --config scribe-cluster.toml df
scribe-ctl --config scribe-cluster.toml status --storage

text

Scribe · local · storage inventory

  SERVER     REPOS          USED     AVAILABLE  ENDPOINT
  storage-a      2      2.80 MiB    436.11 GiB  tls://127.0.0.1:7447
  storage-b      1      2.25 MiB    436.11 GiB  tls://127.0.0.1:7448

  Total: 3 repositories · 5.05 MiB used

df asks every store directly, as the scribe-monitor service account whose credential add auth issued; repo list (stage 3) adds each repository, and status --storage combines both with the service table. These are live samples; --json gives exact byte counts for scripts.

4. Who can do what

scribe-ctl --config scribe-cluster.toml access --user bob

text

Scribe · local · access
  ACCOUNT  KIND    STATE    SCOPE                                               ACCESS       DETAIL
  bob      person  enabled  repository studio/689a965e479c1decf32f3c9ca7e3f913  writer
  bob      person  enabled  sessions                                            2 signed in  latest expires 2026-12-26

Bob has two sessions: one for storage A from stage 2 and one for the gateway from stage 3. Without --user, access lists every account (stage 2 showed it). It reads through the administrator credential and needs auth running. To change access, use scribe-auth as in stage 2. To create accounts from a file instead, see accounts and pepper.

5. Renew credentials and certificates

Every service credential (storage-a, storage-b, the gateway's, scribe-monitor's) and every service's TLS certificate is valid for 365 days. renew replaces each one that expires within 30 days, and does nothing otherwise:

scribe-ctl --config scribe-cluster.toml renew --plan
scribe-ctl --config scribe-cluster.toml renew

text

Scribe · local · renew plan · gateway variant · native runtime
  ...
  CHECKS
    ok    credentials   nothing to renew: 5 credentials are valid for 30 more days; the first to expire is the storage-a server credential on 2027-09-27
    ok    certificates  4 server certificates are valid for 30 more days; the first to expire is auth's on 2027-09-27
    ok    cluster       running; no service needs a restart
  ...
Scribe · local · Unchanged: nothing to renew: 5 credentials are valid for 30 more days; the first to expire is the storage-a server credential on 2027-09-27
  4 server certificates are valid for 30 more days; the first to expire is auth's on 2027-09-27

When something is due, renew issues the replacement, puts it in place, restarts the services that read it, proves it works, and only then revokes the old one; if a proof fails, the old credentials go back and it exits 1. Run it daily from a scheduler: renew service credentials has cron, systemd and Task Scheduler examples.

Server certificates are reissued for the key they already have, by the CA that signed them, so clients notice nothing. That needs the CA's private key: ca-key.pem beside the cluster, or, once you have moved the keys offline as where advises, renew --ca-dir DIR (storage and gateway CA) and --auth-ca-dir DIR (the auth CA) pointing at them.
The administrator credential setup wrote never expires. Once the definition names an expiring one (see administrator credentials), renew replaces it too.
The web interface's own certificate (stage 5) stays yours.

Rotate a CA

The two server-certificate CAs are valid for 10 years; status, doctor and renew say so a year ahead. A rotation replaces one without asking any client to trust something by hand, as long as the client connects during an overlap. It is never started by a scheduled renew:

scribe-ctl --config scribe-cluster.toml renew --rotate-ca storage --plan
scribe-ctl --config scribe-cluster.toml renew --rotate-ca storage

text

  ...
  ROTATION
    The new storage and gateway CA is current: BLAKE3:68bf8bcb6e2c0679ca33473f8dd705b5c83f12c9ff82a0ce21bcfa61f0904991
    The old one (BLAKE3:26e411692a534f4d68414ab4226585414fa41a489382371fd9bc8be07f90bd7b) stays trusted through the cross certificate for 90 days; a plain renew then retires it.
    Clients learn the new CA when they connect in that time; give new clients the new ca.pem.

Both stores and the gateway now present their certificate followed by a handover certificate the old CA signed for the new one. A client that trusts only the old CA verifies that chain and saves the new CA by itself (scribe trust show then lists both); the next scheduled renew after the overlap retires the old CA, and clients drop it. status shows the day:

text

  CA  the storage and gateway CA is being rotated since 2026-09-27; a plain renew retires the old CA on 2026-12-26 (`scribe-ctl --config /home/you/scribe-demo/scribe-cluster.toml renew --retire-ca storage` does it now)

--rotate-ca auth rotates the auth CA the same way, and --rotate-ca credential|access auth's two internal CAs. What happens to clients is in when a server's CA changes; every CA and how to replace each without scribe-ctl is in certificates and credentials.

6. Upgrade to a new build

After pulling new source into your checkout, one line from the checkout builds it, upgrades the cluster, and installs the new executables only if the upgrade succeeded. Run it from the checkout, with the path of your scribe-demo directory. Look at the plan first (-- --plan passes --plan to scribe-ctl upgrade; nothing is installed):

./scripts/install.sh --upgrade --config ~/scribe-demo/scribe-cluster.toml -- --plan
./scripts/install.sh --upgrade --config ~/scribe-demo/scribe-cluster.toml

text

Scribe · building release binaries (build tutorial-next)
...
Scribe · local · upgrade plan · gateway variant · native runtime
  ...
  CHECKS
    ok    release         0.1.0 (build 86da462a7d46.dirty-20260927T124201) → 0.1.0 (build tutorial-next): the same version, another build
    ok    installed set   0.1.0 (build 86da462a7d46.dirty-20260927T124201) in cluster-state/local/bin/86da462a7d46.dirty-20260927T124201
    ok    new set         /home/you/scribe/target/install/aarch64-apple-darwin/release: scribe-ctl, scribe-auth, scribe-server, scribe-gateway 0.1.0 (build tutorial-next) for macos/aarch64, and the scribe client (not installed)
    ok    controller      scribe-ctl 0.1.0 (build tutorial-next)
    ok    formats         unchanged: every executable reads and writes what the active set does
    ok    cluster         running; drained after the new set is installed
    ok    auth data       auth_schema 4; no migration: its formats do not change (no dry run: auth serves it and holds its data-root lock)
    ok    storage-a data  catalog_schema 1, envelope 1, object_store 1, repository_schema 3; no migration: its formats do not change (no dry run: storage-a serves it and holds its data-root lock)
    ok    storage-b data  catalog_schema 1, envelope 1, object_store 1, repository_schema 3; no migration: its formats do not change (no dry run: storage-b serves it and holds its data-root lock)
    ok    gateway data    route_store_schema 2; no migration

  STEPS
     1. Install scribe-ctl, scribe-auth, scribe-server, scribe-gateway 0.1.0 (build tutorial-next) side by side in cluster-state/local/bin/tutorial-next (from /home/you/scribe/target/install/aarch64-apple-darwin/release)
        side by side with the active set, before anything stops
     2. Drain the running cluster
        roots are backed up, migrated and switched only while nothing holds them
     3. Switch the active set from 86da462a7d46.dirty-20260927T124201 to tutorial-next (bin/current)
        services start from the set bin/current names
     4. Start the cluster with tutorial-next and wait until every service is ready
        the health check: every service must become ready
     5. Write the cluster summary CLUSTER-INFO.txt
        the summary names the active set

  ROLLBACK (if step 3 or a later one fails)
     6. Drain whatever of the cluster runs
     ...
Plan only: the installed executables are unchanged.

The build id names each build (the checkout's commit, with .dirty-<time> for local changes, or SCRIBE_BUILD_ID when you set it; this run set SCRIBE_BUILD_ID=tutorial-next). The real run:

text

Scribe · local · Upgrade complete
  installed 0.1.0 (build tutorial-next) in cluster-state/local/bin/tutorial-next
  drained the cluster
  switched the active set from 86da462a7d46.dirty-20260927T124201 to tutorial-next
  every service is ready, running tutorial-next
  wrote CLUSTER-INFO.txt
  ...
  Installed  /home/you/.local/bin/scribe
  Installed  /home/you/.local/bin/scribe-server
  ...
Installation complete. No profiles changed.

What scribe-ctl did for you

The new release's own scribe-ctl ran the upgrade, because only it knows that release's migrations. It checked that the new executables are one complete release for this machine and not older than the running one, compared the formats each reads and writes, and ran migration dry runs where it could. It installed the new set beside the old one while the cluster kept serving, then drained it, switched bin/current, started the new set and waited until every service was ready. When a format changes, it also backs up each root a migration changes before migrating it, and if anything fails once the cluster is drained, it rolls back by itself: old executables, restored roots, the cluster running as before. Nothing is deleted: where lists the earlier set, kept until you remove it. See upgrade to a new release.

With a downloaded release instead of a checkout, unpack it into a directory of its own and give that to scribe-ctl directly:

scribe-ctl --config scribe-cluster.toml upgrade --binaries /path/to/release --plan
scribe-ctl --config scribe-cluster.toml upgrade --binaries /path/to/release

A newer scribe-ctl in that directory takes the upgrade over; use it (or copy it where you keep scribe-ctl) from then on.

Check that clients still work: (cd work/alice && alice sync) and alice repo list show the same repositories.

7. Stop, or run attached

scribe-ctl --config scribe-cluster.toml down
scribe-ctl --config scribe-cluster.toml up --foreground

down drains the gateway, then storage, then auth, and never force-kills a durable commit. up --foreground keeps the supervisor in your terminal (overriding daemon = true); Ctrl-C requests the same graceful shutdown. Neither arranges startup after a reboot or restarts a failed service; see production deployment.

8. Operate a cluster on another machine, or take one over

Everything in this stage works on a cluster on another machine too: name it in the definition's [remote] table (or pass --remote deploy@scribe.example.com:/srv/scribe), and every command runs there over the system ssh, with the definition here as the source of truth and the secrets staying there. scribe-ctl ssh opens a shell in the cluster's directory. See remote operation.

A Docker deployment that scribe-ctl did not create (Compose projects you run by hand, for example these) comes under the controller with adopt, without moving its data or secrets: see take over an existing Docker deployment. Native services you started by hand are brought under it with a definition that names their files (the definition file).

Continue with 5. Start the web interface. For ongoing care, see the operations guide and the maintenance guide; for the same operations without scribe-ctl, see certificates and credentials and upgrades by hand.