Certificates and credentials

Previous: web interface · Manual setup · Next: upgrades

Everything a Scribe deployment uses to prove who it is expires: service credentials, server TLS certificates, the certificate authorities that sign them, and people's sessions. This guide says what each one is, when it expires, and how to replace it before it does, with scribe-ctl and by hand. Clients' side of a CA change is in client trust.

The by-hand commands are shown for a Unix shell and, where they differ, for PowerShell on Windows. They assume the layout of the bare-metal guide in /srv/scribe (C:\Scribe); for the Docker Compose layout, see below.

The four certificate authorities

flowchart LR
    subgraph Files["Server CAs: files you keep (pki/)"]
        CA["Storage and gateway CA<br/>pki/ca"]
        ACA["Auth CA<br/>pki/auth-ca"]
    end
    subgraph DB["Internal CAs: inside the auth database"]
        CRED["Credential CA"]
        ACC["Access CA"]
    end
    CA --> SC["Storage and gateway<br/>TLS certificates"]
    ACA --> AC["Auth's TLS certificate"]
    CRED --> CC["Credential certificates<br/>(sessions, credential files)"]
    ACC --> XC["Access certificates<br/>(one hour)"]

Every CA key and server key Scribe creates is ML-DSA-65, a post-quantum signature (see shared authentication). A CA created before that, with an ECDSA P-256 key, keeps signing until you rotate it (scribe-ctl doctor warns about it until then); its successor is ML-DSA-65, and the handover carries clients' trust across as for any rotation.

The two server CAs are ordinary files: ca.pem is public, ca-key.pem signs and belongs offline once the certificates you need exist. They are created with a path length of 1, so exactly one CA may sit below them: the handover certificate of a rotation, which strict validators (OpenSSL, Node) then accept. A CA created before that, with path length 0, still rotates; the web services are the strict validator that would refuse its handover chain, and scribe-ctl gives them both CAs for the overlap so they never need the chain. The internal CAs never leave the auth database and no client sees them; how they are used is in identity, device keys and certificates.

What expires

scribe-auth access shows every account's credentials with their expiry; scribe-ctl status, where and doctor warn 30 days before a credential or server certificate expires, and a year before a CA does. doctor also warns while a CA still has a classical key (ca.key.<storage|auth>, ca.internal.<credential|access>.key) and names the rotation that replaces it.

With scribe-ctl

scribe-ctl renew replaces every credential and server certificate the cluster owns that expires within 30 days, and only then revokes the old one; run it daily from a scheduler (renew service credentials has the details, the CA key it needs, and cron, systemd and Task Scheduler entries):

scribe-ctl --config /srv/scribe/scribe-cluster.toml renew --plan
scribe-ctl --config /srv/scribe/scribe-cluster.toml renew

Credentials scribe-ctl does not manage (machines, CI, administrators elsewhere) are replaced by hand.

Rotate a server CA with a handover

A rotation replaces a server CA without asking any client to trust something by hand, as long as the client connects during an overlap. It is only started by you, for the storage and gateway CA or for the auth CA:

scribe-ctl --config /srv/scribe/scribe-cluster.toml renew --rotate-ca storage --plan
scribe-ctl --config /srv/scribe/scribe-cluster.toml renew --rotate-ca storage --overlap-days 90

  ROTATION
    The new storage and gateway CA is current: BLAKE3:68bf8bcb6e2c0679ca33473f8dd705b5c83f12c9ff82a0ce21bcfa61f0904991
    The old one (BLAKE3:26e411692a534f4d68414ab4226585414fa41a489382371fd9bc8be07f90bd7b) stays trusted through the cross certificate for 90 days; a plain renew then retires it.
    Clients learn the new CA when they connect in that time; give new clients the new ca.pem.

After the overlap a scheduled plain renew retires the old CA; renew --retire-ca storage does it early. How the handover works, what --rotate-ca auth also rewrites, and the rollback are in rotate a CA. Credential files scribe-ctl does not manage carry the auth CA too and need new ones after an auth CA rotation retires.

Without scribe-ctl there is no command that makes a handover certificate: change the CA by hand instead, and clients trust the new one explicitly.

Rotate auth's internal CAs

scribe-ctl renew --rotate-ca credential|access adds a successor; a plain renew retires the oldest once the overlap ends, and renew --retire-ca credential|access does it now (rotate a CA). By hand, see below.

By hand

Before you change anything

• Find the subject, its credential IDs and expiry with auth access --user NAME and auth credentials list --user NAME (server credentials: auth credentials list, subject in the third column). Know every consumer of the file.
• Keep one working administrator credential throughout.
• A credential is a device key and the certificate auth issued for it, in one credential file. Replacing one always means a new file with a new key: issue it, install it, restart its holder if it reads the file only at startup, verify, and only then revoke the old one. credentials create never overwrites a file, so issue to a new name.
• Preserve any pending administrative request until it is resolved: repeating the same command resolves it (administration and retries).

The auth helper of the bare-metal guide is used throughout.

In the Docker Compose layout

In the Docker Compose guide's project in /srv/scribe, the same steps run through the project's services:

• auth X is docker compose run --rm auth-admin X. It writes new files to /new/; move each from secrets/new/ into the directory its service mounts (secrets/storage-a/, secrets/gateway/, secrets/admin/). A file the command reads, such as a new --auth-ca, goes in with a mount of its own: docker compose run --rm -v ./pki/auth-ca-2/ca.pem:/auth-ca-2.pem:ro auth-admin credentials create … --auth-ca /auth-ca-2.pem.

• certs init-ca and certs issue run through the auth image with the whole pki/ mounted; the mount replaces the service's own /pki for that run:

  docker compose run --rm --no-deps -v ./pki:/pki auth certs issue --ca /pki/ca --out /pki/storage-a-2027 --dns storage-a --dns localhost --ip 127.0.0.1
  docker compose run --rm --no-deps -v ./pki:/pki auth certs init-ca --out /pki/ca-2

• Offline auth commands (authorities, admin-credential issue) run through the stopped auth service, whose data directory is /data and whose witness is /witness/auth.revision:

  docker compose stop auth
  docker compose run --rm --no-deps auth authorities list --data-dir /data
  docker compose run --rm --no-deps auth authorities retire --data-dir /data --purpose access --revision-file /witness/auth.revision
  docker compose run --rm --no-deps -v ./secrets/admin:/admin -v ./pki/auth-ca/ca.pem:/auth-ca.pem:ro \
    auth admin-credential issue --data-dir /data --user admin --out /admin/admin-recovery.credential \
    --auth tls://auth:7443 --auth-ca /auth-ca.pem
  docker compose up -d --wait

• Restarting. A service reads a file replaced inside a directory it mounts after docker compose restart SERVICE. A new certificate or CA directory is a new mount source: change that mount's source: in compose.yaml (a service's /pki, the gateway's /trust/ca.pem) and run docker compose up -d --wait, which recreates the services whose definition changed. docker compose restart does not read compose.yaml again.

New credential files are written private (mode 0600 on Unix, an access list for the writing account, SYSTEM and Administrators on Windows). Scribe refuses one that other accounts can read, or whose directory lets them replace it. To repair a file's permissions:

chmod 600 /srv/scribe/secrets/storage-a.credential
chmod 700 /srv/scribe/secrets

icacls C:\Scribe\secrets /inheritance:r /grant:r "${env:USERNAME}:(OI)(CI)F" "SYSTEM:(OI)(CI)F"

Storage server and gateway credentials

Both read their credential only at startup. Issue the replacement beside the current file:

auth credentials create storage-a --server --name daemon --days 365 --out ./secrets/storage-a.credential.next
auth credentials create scribe-gateway --service --name gateway --days 365 --out ./secrets/gateway.credential.next

Keep the current file, move the new one into its place, and restart the service (the gateway keeps its namespace reader and delegate grants):

mv secrets/storage-a.credential secrets/storage-a.credential.previous
mv secrets/storage-a.credential.next secrets/storage-a.credential
touch run/storage-a/shutdown-request   # then start it again, or restart it under your service manager

Move-Item secrets\storage-a.credential secrets\storage-a.credential.previous
Move-Item secrets\storage-a.credential.next secrets\storage-a.credential
New-Item -ItemType File run\storage-a\shutdown-request | Out-Null

Verify that a client can list and sync an authorized repository (for the gateway, that scribe-gateway status shows fresh inventory from every instance), then revoke the old credential and delete the kept file:

auth credentials list | grep storage-a
auth credentials revoke OLD_CREDENTIAL_ID
rm secrets/storage-a.credential.previous

auth credentials list | Select-String storage-a
auth credentials revoke OLD_CREDENTIAL_ID
Remove-Item secrets\storage-a.credential.previous

If verification fails, move the kept file back, restart, and revoke the new credential instead. In Docker, a container that mounts the file alone (as scribe-ctl generates them) keeps the old file after a rename; overwrite it in place instead, or mount its directory as the Docker Compose guide does.

Service-account and web reader credentials

Machines, CI jobs, the inventory account and the web services read their credential file on every use, so no restart is needed: issue a new file, point the holder at it (or rename it over the old one), verify one command, and revoke the old credential.

auth credentials create build-farm --service --name ci-read --days 90 --out ./secrets/build-farm.credential.next
SCRIBE_CREDENTIAL_FILE=/srv/scribe/secrets/build-farm.credential.next scribe repo list

auth credentials create build-farm --service --name ci-read --days 90 --out .\secrets\build-farm.credential.next
$env:SCRIBE_CREDENTIAL_FILE = 'C:\Scribe\secrets\build-farm.credential.next'
scribe repo list

For the web reader, issue it with the address the web services reach auth at (--auth, and --auth-ca when they verify auth with another CA; see the web guide), rename it over deploy/local/worker/reader.credential in one step, and give uid 1000 read access again if you used an ACL on Linux. A lapsed reader credential stops every indexing job until a new one is in place.

Administrator credentials

The bootstrap credential init wrote never expires. Every other administrator credential does, so keep either a second administrator credential sealed offline or the ability to run the offline recovery on the auth host.

To replace the bootstrap credential, or renew an expiring administrator credential, issue one for the same administrator (or another auth administrator, created with auth users add NAME --administrator), use it, then revoke the old one with the new one:

auth credentials create admin --name admin-2027 --days 3650 --out ./secrets/admin-2027.credential
scribe-auth --admin-credential ./secrets/admin-2027.credential --state-dir ./admin-state users list

Point the auth helper (and [auth] admin_credential of a scribe-ctl definition) at the new file, then:

auth credentials list --user admin
auth credentials revoke OLD_CREDENTIAL_ID

The new file names the same auth address and CA as the one that created it; give --auth and --auth-ca when its holder reaches auth elsewhere. Auth refuses to revoke the last live administrator credential, and a password session never administers.

Server TLS certificates

By hand, issue a new certificate from the same CA into a new directory, with the names clients use:

scribe-server certs issue --ca ./pki/ca --out ./pki/storage-a-2027 --dns scribe.example.com --days 365

certificate: /srv/scribe/pki/storage-a-2027/server.pem
private key: /srv/scribe/pki/storage-a-2027/server-key.pem
client trust CA: /srv/scribe/pki/storage-a-2027/ca.pem
CA fingerprint: BLAKE3:bec9ed90312d4b568ddb7752e5a26e6e80789e9b724b23b20a25a5a86e1c82f9

The fingerprint is the CA's, unchanged, so clients notice nothing. certs issue writes a new key as well; scribe-ctl renew instead reissues for the existing key, and scribe-ctl renew --rekey SERVICE does what this section does, a new key and certificate from the same CA, proved before the old key goes (replace server keys). Point tls_cert and tls_key (the gateway's [tls] cert and key, or auth's --tls-cert and --tls-key) at the new files, restart the service, check a client connection by its real name, and remove the old key. Use scribe-auth certs issue --ca ./pki/auth-ca for auth's certificate. The CA must stay valid for the whole new lifetime; its key goes back offline.

Change a server CA by hand

Without a handover, every client that trusts the old CA is refused once the servers present the new one, and has to trust the new CA explicitly. Plan a window, and publish the new fingerprint through a channel your users trust.

Storage and gateway CA. Create the new CA and a certificate from it for every storage server and the gateway:

scribe-server certs init-ca --out ./pki/ca-2
scribe-server certs issue --ca ./pki/ca-2 --out ./pki/storage-a-ca2 --dns scribe.example.com
scribe-server certs issue --ca ./pki/ca-2 --out ./pki/gateway-ca2 --dns scribe.example.com

Point each service's certificate and key at the new directories, point every gateway instance's ca (and any scribe-ctl or web trust file) at pki/ca-2/ca.pem, and restart the gateway, then the stores. A client then reports:

scribe: The CA of tls://127.0.0.1:7447 changed.
  trusted: BLAKE3:bec9ed90312d4b568ddb7752e5a26e6e80789e9b724b23b20a25a5a86e1c82f9
  now:     BLAKE3:2462fcd2d18e01d3ecc2c0d88d152dc1f77ecccc52332b7debb6c36c6501dfec
If your administrator confirms this change, run: scribe --server tls://127.0.0.1:7447 trust replace

and trusts the new CA with trust replace (client side); clients that pass --tls-ca need the new file.

Auth CA. Every credential file carries the auth CA, so issue the new files first, while auth still presents its current certificate. Create the CA and auth's certificate, then a new credential for every holder, naming the new CA:

scribe-auth certs init-ca --out ./pki/auth-ca-2
scribe-auth certs issue --ca ./pki/auth-ca-2 --out ./pki/auth-ca2 --dns auth.example.com
auth credentials create admin --name admin-2 --days 3650 --auth tls://auth.example.com:7443 --auth-ca ./pki/auth-ca-2/ca.pem --out ./secrets/admin-2.credential
auth credentials create storage-a --server --name daemon-2 --days 365 --auth tls://auth.example.com:7443 --auth-ca ./pki/auth-ca-2/ca.pem --out ./secrets/storage-a-2.credential
auth credentials create scribe-gateway --service --name gateway-2 --days 365 --auth tls://auth.example.com:7443 --auth-ca ./pki/auth-ca-2/ca.pem --out ./secrets/gateway-2.credential

Repeat for every storage server, service account and machine. With a web interface, issue its reader credential the same way, with the address the web services reach auth at (web guide, step 3), and put both CAs in its trust file for the switch, then restart the web services:

auth credentials create scribe-web --service --name web-reader-2 --days 365 --auth tls://auth.example.com:7443 --auth-ca ./pki/auth-ca-2/ca.pem --out ./secrets/web-reader-2.credential
cat ./pki/auth-ca/ca.pem ./pki/auth-ca-2/ca.pem > /path/to/scribe/apps/web/deploy/local/trust/auth-ca.pem

Then stop the gateway, the stores and auth; start auth with the new certificate; point each service at its new credential file; start the stores and the gateway. Rename the new reader credential over deploy/local/worker/reader.credential (the web services read it on every use). The old administrator credential no longer verifies auth (invalid peer certificate: BadSignature): point the auth helper (and [auth] admin_credential of a scribe-ctl definition; in the Docker Compose layout, the file in secrets/admin/) at secrets/admin-2.credential, and with it revoke every old credential. Once everything works, leave only the new CA in the web interface's auth-ca.pem and restart the web services again.

People who approved auth's certificate with trust add (shared trust) trust the new CA once, naming auth's own address: scribe --server tls://auth.example.com:7443 trust replace. People who logged in with --auth-ca replace the copy saved with each session instead, once for each server they have a session for: scribe --server SERVER trust replace --auth-ca FILE (client side).

Auth's internal CAs

The credential and access CAs live in the auth database. Rotate one with auth stopped (the commands take its lock); a successor signs 15 minutes after it is added, by when every service has fetched it:

scribe-auth authorities list --data-dir ./data/auth
scribe-auth authorities add --data-dir ./data/auth --purpose access

credential 0000	CN=Scribe credential CA	BLAKE3:c97307433a50dac0875dc59705e49d7592ba1def96a5577b006c53611b734a36	expires 2046-09-27	signing
access 0000	CN=Scribe access CA	BLAKE3:17ab45dae8663ab81f10eb6cd2493e0f4971d05b5b92f05c0c0bc526dba22a26	expires 2046-09-27	signing
access 0001	CN=Scribe access CA 1	BLAKE3:6b651cec197b1000f173bf7f0f9a1df7288f4a41d336c8e3ac3d6916042195b7	expires 2046-09-27	signs from 2026-09-27T22:42:34.000Z

Start auth again. Retire the oldest once nothing it signed can still be presented: for the access CA an hour after its successor began signing, for the credential CA once every live credential was renewed from the new one (auth names the ones left; the bootstrap administrator credential never expires, so replace it). With --revoke-held, the credential CA's retirement revokes the sessions and credentials still holding one of its certificates instead (their holders sign in again), never every administrator credential. A retirement moves the revision, so it needs the witness:

scribe-auth authorities retire --data-dir ./data/auth --purpose access --revision-file ./secrets/auth.revision

scribe-auth: the oldest access CA cannot be retired yet: its successor signs only from 2026-09-27T22:42:34.000Z; retire after that

add refuses while an earlier successor has not retired its predecessor. Both are audited (authority-add-offline, authority-retire-offline).

Lost, expired or compromised

Revoke a compromised credential at once rather than waiting for a routine window: revocation affects new storage admissions within 30 seconds, whatever the certificate's lifetime; already admitted requests may finish.

No administrator credential left

Stop scribe-auth serve, back up the auth data directory, and issue a short recovery credential directly into it for an existing, enabled auth administrator:

scribe-auth admin-credential issue --data-dir ./data/auth --user admin \
  --out ./secrets/admin-recovery.credential \
  --auth tls://auth.example.com:7443 --auth-ca ./pki/auth-ca/ca.pem

issued an offline administrator credential for admin; saved to ./secrets/admin-recovery.credential; credential 7b48b9d36a061fe2f3b21f717e9f0e73; expires 1790635579999 (Unix ms)

It lives one day by default (--days), is labelled offline-recovery and is audited as admin-credential-issue-offline. It never creates an administrator or changes an account, and never overwrites a file. Start auth, issue a regular administrator credential with it (as above), and revoke the recovery credential. Never rerun init or edit the database.