# Production deployment checklist

Use this after the [tutorials](/docs/tutorials.md) or the [manual setup
guides](/docs/manual.md), before opening access to a team.

## 1. Choose the topology

| Need | Deployment |
| --- | --- |
| One shared project service | Auth + one storage instance |
| Several storage instances behind one address | Auth + storage instances + gateway |
| Manage services on one host | Add ctl, or manage each service with the host's service manager ([by hand](/docs/manual/bare-metal.md#run-them-as-services)) |

> [!NOTE]
> A gateway scales placement across repositories. It does not shard a single repository, rebalance existing data, or provide replication/failover. Ctl supervises the processes of one machine, the cluster's own; it drives that machine from another one over ssh ([remote operation](/docs/cluster-usage.md#remote-operation)), but does not spread one cluster across hosts.

Choose stable DNS names and ports. Use matching client and service revisions.

## 2. Set ownership and storage paths

- Use unprivileged service accounts with access only to their own resources.
- Give each instance distinct data and runtime roots; keep logs separate.
- Keep secrets (credential files, TLS keys, the password pepper, the auth
  revision witness) outside data roots and readable only by their consumers.
  On Unix, Scribe refuses a credential file, device key, pepper, or witness
  that grants any access to group or other accounts, or whose directory they
  can write. On Windows it refuses one whose access list allows any account
  other than its owner, SYSTEM and Administrators more than reading its
  attributes, or whose directory lets another account delete its entries.
- Keep runtime directories private: services refuse one that group or other
  accounts can write.
- Keep the CA signing key offline. Clients receive public trust material and
  their own device keys and certificates, never service keys or administrator
  credentials.

Ctl runs its children under its account. For separate service identities,
supervise each service independently.

Check [storage durability assumptions](/docs/durability-and-recovery.md#assumptions)
and [capacity needs](/docs/operations.md#capacity-planning), including retained history,
uploads, backups, and client recovery files.

## 3. Expose only the required connections

| Connection | Purpose |
| --- | --- |
| Clients → storage or gateway | Repository operations over mutual TLS |
| Clients → auth | Login, logout, and access-certificate renewal over TLS |
| Storage → auth | Identity and permission checks with the server's credential |
| Gateway → storage | Requests and inventory over mutual TLS |
| Gateway → auth | Access CAs and access certificates with the gateway's credential |
| Operator tools → auth | Administration with an administrator credential |

Every Scribe endpoint requires TLS 1.3 with the hybrid post-quantum key
exchange `X25519MLKEM768`. Load balancers and proxies in these paths must pass
TLS through: one that terminates TLS cannot present the client's certificate
and usually cannot offer that key exchange. Restrict listeners with host/network rules. Behind a gateway, allow backend
access only where needed by the gateway and operators. Keep runtime files private.

> [!WARNING]
> Do not expose plaintext storage to an untrusted network. `--allow-non-loopback` acknowledges exposure; it does not add authentication or encryption.

Use certificates for real connection names and establish verified trust first.
Review [roles](/docs/auth.md#credentials-and-roles) and schedule
[credential renewal](/docs/manual/certificates.md).

## 4. Provision once, then configure automatic startup

Complete initialization and a client round trip first. Then configure the host
service manager to run ctl **in the foreground**, using your absolute paths:

```sh
/opt/scribe/bin/scribe-ctl --config /etc/scribe/cluster.toml --cluster production up --foreground
```

A service definition meets these requirements:

| Setting | Requirement |
| --- | --- |
| Executable/configuration | Absolute paths; matching service binaries beside ctl or configured explicitly |
| Account | The prepared service account, with access to required files |
| Startup | After required storage mounts and network availability |
| Readiness | Check service state and, for a gateway, fresh backend inventory |
| Shutdown | Deliver graceful termination and allow started durable work to finish; avoid a short forced-kill deadline |
| Restart | An explicit operator-selected policy; investigate repeated failures before retrying indefinitely |

Ctl daemon mode alone does not start at boot. Give each process one supervisor.
See [lifecycle behavior](/docs/cluster-usage.md#lifecycle-and-diagnostics).

### systemd

For a native ctl cluster, one unit runs the supervisor. SIGTERM makes ctl
drain the services in order (gateway, storage, auth); `KillMode=mixed` sends
it to ctl alone, so the services are not signalled all at once, and the long
stop timeout lets started durable work finish:

```ini
# /etc/systemd/system/scribe.service
[Unit]
Description=Scribe cluster
Wants=network-online.target
After=network-online.target local-fs.target

[Service]
Type=simple
User=scribe
Group=scribe
UMask=0077
ExecStart=/opt/scribe/bin/scribe-ctl --config /etc/scribe/cluster.toml --cluster production up --foreground
KillSignal=SIGTERM
KillMode=mixed
TimeoutStopSec=15min
Restart=on-failure
RestartSec=30s

[Install]
WantedBy=multi-user.target
```

Without ctl ([bare metal](/docs/manual/bare-metal.md#run-them-as-services)), give
each service a unit of its own and order them: **auth → storage → gateway**
at start, and systemd stops them in reverse. For storage A of the bare-metal
layout:

```ini
# /etc/systemd/system/scribe-storage-a.service
[Unit]
Description=Scribe storage A
Wants=network-online.target scribe-auth.service
After=network-online.target scribe-auth.service

[Service]
Type=simple
User=scribe
Group=scribe
UMask=0077
WorkingDirectory=/srv/scribe
ExecStart=/opt/scribe/bin/scribe-server serve --config /srv/scribe/storage-a.toml --runtime-dir /srv/scribe/run/storage-a --log-dir /srv/scribe/logs/storage-a
KillSignal=SIGTERM
TimeoutStopSec=15min
Restart=on-failure
RestartSec=30s

[Install]
WantedBy=multi-user.target
```

`scribe-auth.service` runs the `scribe-auth serve` command of [stage
2](/docs/manual/bare-metal.md#initialize-and-start-auth), and
`scribe-gateway.service` runs `scribe-gateway --config
/srv/scribe/scribe-gateway.toml serve --runtime-dir /srv/scribe/run/gateway`
with `After=` and `Wants=` naming every storage unit. Enable them with
`systemctl enable --now scribe-auth scribe-storage-a scribe-gateway`. Every
service treats SIGTERM as a graceful stop ([shutdown](/docs/operations.md#shutdown)).

### Windows

`scribe-server`, `scribe-auth`, `scribe-gateway` and `scribe-ctl` have no
Windows service entry (only [`scribe-runner`](/docs/reference/cli/scribe-runner.md#service-install)
does), so `sc.exe` cannot run them directly, and Scribe documents no service
wrapper for them. Run them in a signed-in session or through a wrapper of
your choice, and stop each one gracefully with its `shutdown-request` file
rather than by killing the process.

### Docker

A Compose project, one you wrote ([Docker
Compose](/docs/manual/docker-compose.md#operate-the-project)) or a Docker-runtime
ctl cluster, comes back after a reboot through `restart: unless-stopped` once
the Docker engine starts at boot (`systemctl enable docker`); do not also run
`scribe-ctl up --foreground` from a service manager, which would stop the
project whenever it stops ([`up`](/docs/reference/cli/scribe-ctl.md#up)). A project
stopped with `docker compose stop` or `scribe-ctl down` stays stopped.

## 5. Assign monitoring and backup responsibilities

| Area | Operational check |
| --- | --- |
| Availability | Service exits, failed logins, backend health, stale inventory |
| Capacity | Free disk space and growth of history, logs, and recovery files |
| Credentials | Server TLS certificate and machine credential expiries; protected access to the original pepper and revision witness |
| Logs | Set service retention; plan separate retention for ctl/supervisor logs, which ctl does not rotate |
| Integrity | Schedule offline doctor checks, particularly before backup and after unclean shutdown |

Back up everything [what to back up](/docs/migration-backup-restore.md#what-to-back-up)
lists: each storage root, the stopped auth database, the stopped gateway
ledger and, with the web interface, its database, object store and secrets. Keep the matching pepper in a separate
protected backup, and keep the revision witness out of the auth data backup.
Outside an [upgrade rollback](/docs/migration-backup-restore.md#upgrade-rollback-the-one-exception),
restore auth only with [`scribe-auth restore`](/docs/reference/cli/scribe-auth.md#restore)
([auth backup and restore](/docs/migration-backup-restore.md#auth-backup-and-restore)):
storage servers refuse an auth copied back from an older backup. Preserve
configuration, credentials/keys, and ctl provisioning state.

Use independent backup storage and rehearse recovery. Schedule maintenance windows
for offline checks and copies; rotate ctl log files during a planned stop.

## 6. Verify before opening access

- [ ] A writer can log in, create a working copy, submit, and sync through the intended endpoint.
- [ ] A reader can sync but cannot submit; an ungranted user cannot discover the repository.
- [ ] A graceful stop/start preserves repository IDs, history, and access.
- [ ] Gateway inventory is healthy and existing repositories keep their assignments.
- [ ] Backups, expiry monitoring, log retention, and an operator owner are in place.

Use [troubleshooting](/docs/troubleshooting.md) for failed checks.