# Server setup and maintenance For every command and option, see the [server CLI reference](/docs/reference/cli/scribe-server.md). This guide covers configuration and service behavior. To set a server up, use `scribe-ctl` ([tutorial 1](/docs/tutorials/01-minimal-server.md), then [add authentication](/docs/tutorials/02-add-auth.md)) or run it by hand on [bare metal](/docs/manual/bare-metal.md) or with [Docker Compose](/docs/manual/docker-compose.md). All command-line tools support global `--json`; see [JSON output for automation](/docs/json-output.md) for results, errors, and exit codes. Logging and JSON switches are listed in the [CLI reference](/docs/reference/cli/scribe-server.md#global-options). See [logging](/docs/logging.md) for filters, file retention, and timings. Offline maintenance commands that open a data root take the same exclusive lock as `serve`, so they refuse to run beside a live server. Most select the root with `--config FILE` or `--data-dir DIR`; `auth-init`, `repair-objects`, `recovery-archive`, `compact-objects`, and `recover-submissions` accept only `--data-dir DIR`, and `restore` and `restore-checkpoint` take source and destination directories instead. Two commands work beside a running server and take no data-root lock: `checkpoint` asks the server that owns `--runtime-dir` to take an online backup, and `integrity-status` reads saved verification reports from the runtime directory without opening the data root. Certificate and remote auth-administration commands do not need a data root. ## init [`init`](/docs/reference/cli/scribe-server.md#init) prepares a new deployment explicitly. For a cluster it manages, `scribe-ctl setup` runs it ([first-server tutorial](/docs/tutorials/01-minimal-server.md#2-set-up-the-cluster)). By hand, `init` is followed by [`serve`](/docs/server-usage.md#serve) ([bare metal, stage 1](/docs/manual/bare-metal.md#1-one-storage-server)). Creates an empty data root (descriptor, catalog, `repos/`, lock file) and optionally writes a configuration whose paths are relative to the configuration file. Data-root directories are created private (0700 on Unix). Omit `--config-out` when you only need the data root. Refuses a non-empty data directory, and refuses to overwrite an existing configuration without `--force`. ## serve Use [`serve`](/docs/reference/cli/scribe-server.md#serve) for an initialized root. `SCRIBE_SERVER_CONFIG` may name the configuration file; flags override it, and flag paths are relative to the current directory. Startup opens the descriptor, refuses unknown or newer formats, takes the lock, opens the catalog and every ready repository (discarding interrupted creations), logs every resolved path and limit, binds, writes `/scribe-server.pid` and `/listen` (the bound address, useful with port 0), then accepts connections. Two modes: - **Plaintext TCP** (no `[auth]` table and no `--auth-*`, `--tls-*` options). Anyone who can reach the port can act as anyone. The server refuses a non-loopback listen address unless `--allow-non-loopback` or `allow_non_loopback = true` is set, and logs `server.plaintext_exposed` when it is. A data root prepared with `auth-init` refuses to start this way. - **TLS with shared authentication.** The four `[auth]` settings are required together, and the data root must have been prepared with `auth-init`. The socket then serves TLS 1.3 only, with the hybrid post-quantum key exchange `X25519MLKEM768` and no other; `listen` reports `tls://host:port`. Clients authenticate with mutual TLS, presenting access certificates that auth issued; the server verifies them against the access CAs it fetches from auth and refreshes every five minutes, and until the first fetch succeeds it refuses every presented certificate. Every request is then authorized per repository. Setup is in [bare metal, stage 2](/docs/manual/bare-metal.md#2-add-authentication); the model in [auth.md](/docs/auth.md). The runtime directory is created private (0700 on Unix). An existing runtime directory that group or other accounts can write is refused at startup (`... is writable by other accounts (mode 777); restrict it with chmod go-w`), because its files stop the server and request backups. Stop with SIGINT, SIGTERM, or `touch /shutdown-request`. Admission stops and idle connections close. In-flight requests get up to `drain_timeout_secs` for network I/O; after that, sockets are interrupted and started storage jobs finish before exit. A stalled disk can extend shutdown. After forced termination, acknowledged work remains durable under the [storage assumptions](/docs/durability-and-recovery.md#assumptions). The next start cleans staging and unreferenced pack tails. ## Configuration file ```toml [server] data_dir = "server-data" # relative to this file log_dir = "logs" # omit for stderr runtime_dir = "run" # default /run listen = "tcp://127.0.0.1:7447" allow_non_loopback = false # plaintext mode only; requires a trusted tunnel [logging] filter = "info" # or "warn,scribe_server=info,scribe_objects=debug" format = "text" # or "json" (newline-delimited) keep_files = 14 # daily files retained in log_dir [limits] max_connections = 256 max_connections_per_principal = 64 # concurrent connections one user may hold max_pending_per_source = 16 # connections one address may hold before its first request storage_workers = 16 # blocking mutations/transfers; >=2 query_workers = 4 # reserved read queries; >0; storage+query+handshake <=1024 transfer_workers = 8 # >0 and 0; never uses query/storage workers staging_budget_mib = 512 # bytes admitted into quarantine across all connections inactivity_timeout_secs = 300 handshake_timeout_secs = 10 drain_timeout_secs = 30 small_object_threshold = 16777236 # envelopes at or below this go to packs pack_seal_mib = 256 max_workspaces_per_principal = 256 # workspace IDs one account may register per repository max_untracked_locks = 4096 # locks on paths that do not exist at head, per repository max_submit_entries = 100000 # change-set entries accepted in one submit; bounds publication gate hold # advertised in server_info; `scribe import` batches a first population to it [integrity] # optional; these are the defaults enabled = true mib_per_second = 1 # decoded verification payload budget idle_secs = 1 # yield to queued/running requests interval_secs = 86400 # wait after each repository pass [auth] # optional; all four keys together namespace = "studio" # registered with scribe-auth servers add --namespace credential = "secrets/storage-a.credential" # scribe-auth credentials create --server tls_cert = "pki/storage-a/server.pem" tls_key = "pki/storage-a/server-key.pem" ``` `scribe-server init --config-out` writes `[server]` and `[logging]` with the values above, `[limits]` and `[integrity]` with every key commented out at its default, and `[auth]` entirely as a comment. Uncomment all four `[auth]` keys together: an `[auth]` table with no keys is refused (`the [auth] table is empty; set namespace, credential, tls_cert and tls_key, or remove the table`), and a partial one fails with `auth requires namespace, credential, TLS certificate and TLS key`. `[auth].credential` is this server's credential file from [`scribe-auth credentials create --server`](/docs/reference/cli/scribe-auth.md#credentials-create). It carries the auth endpoint, the CA of the auth service's TLS certificate, the server's device key, and its credential certificate, so no separate endpoint or CA setting exists. The credential file and `tls_key` are secrets: both must be outside the data directory (`[auth].credential PATH is inside the data directory; keep secrets outside it`), because backups and host moves copy the data root. On Unix the credential file must grant nothing to group or other accounts, in a directory they cannot write; on Windows its access list must allow no other account more than reading its attributes, in a directory where no other account can delete entries. The server reads the credential file only at startup: restart it after replacing the file. See [credential files](/docs/auth.md#credential-files). Limits are validated rather than defaulted: a zero connection, pending, timeout, workspace, lock, or entry limit is refused with `[limits].KEY must be at least 1`, a MiB value that is zero or overflows with `[limits].KEY must be a positive number of MiB that fits in 64 bits`, and an unknown `[logging].format` with `[logging].format must be text or json`. Admission is decided before any backend work: - `max_pending_per_source` bounds the connections one address holds before it has completed its handshake and sent one complete request; an IPv6 source is counted per /64. When every slot is taken, the oldest pending connection is closed to admit a newcomer, so unauthenticated peers can delay a client by at most one handshake deadline. An idle connection counts as pending again until its next request. - `max_connections_per_principal` bounds one user's concurrent connections, decided from the access certificate. Through a gateway the principal is the user it forwards, not the gateway. Beyond it a new connection is refused at its hello with `busy` (`too many concurrent connections for this identity`) and logged as `conn.principal_limit`. Anonymous connections have no principal and remain bounded by the anonymous limits. - One user's request bodies in flight share a quarter of the listener's frame budget (32 MiB of 128 MiB), keyed by the same user id. - New workspace registrations are limited to 64 per user per hour, separately from `max_workspaces_per_principal` (`too many new workspaces this hour; reuse or remove existing workspaces and retry later`, `limit_exceeded`). - An anonymous chunk read of a public repository may make the server read at most 64 MiB of manifests to prove the chunks are public; a larger request is refused (`a public chunk read may name at most 64 MiB of manifests at once; request fewer chunks`). The four worker counts shown are the defaults on a machine with 8 or fewer CPU cores. When a key is omitted, the default scales with the core count `c` (counted up to 64): `storage_workers` is max(2c, 16), `query_workers` is max(c/2, 4), `transfer_workers` is max(c, 8), and `handshake_workers` is max(c/4, 8). The `small_object_threshold` default is the 16 MiB chunk limit plus the 20-byte envelope header, so every chunk-sized envelope is packed. `small_object_threshold` and `pack_seal_mib` are recorded in a repository's object-store descriptor when the repository is created; changing them later affects only repositories created afterward. For background verification, coverage reports, and restart behavior, see [online integrity checking](/docs/online-integrity.md). ## certs Both service executables use the same offline certificate helper. See [certificate rules](/docs/auth.md#generate-a-private-ca-and-server-certificates), the [CLI reference](/docs/reference/cli/scribe-server.md#certs) for options, and [certificates and credentials](/docs/manual/certificates.md) for renewal and CA changes. [Bare metal, stage 2](/docs/manual/bare-metal.md#server-certificates) creates a complete certificate set. ## auth-init, users, permissions Follow [adding authentication](/docs/tutorials/02-add-auth.md) (or, by hand, [bare metal, stage 2](/docs/manual/bare-metal.md#2-add-authentication)) before exposing an existing root to authenticated clients. `auth-init` prepares offline security ownership state; it does not create central users or issue service credentials. The [ownership guide](/docs/auth.md#ownership-and-existing-data) explains why plaintext working copies must be replaced with authenticated working copies. The [`users`](/docs/reference/cli/scribe-server.md#users) and [`permissions`](/docs/reference/cli/scribe-server.md#permissions) conveniences talk to shared auth with an administrator credential file (`--admin-credential`). They use the same implementation as `scribe-auth users` and `scribe-auth grants`. `auth-init` does not register the server at auth: an auth administrator registers it (`servers add`) and issues the credential file named by `[auth].credential` (`credentials create --server`), as in [bare metal, stage 2](/docs/manual/bare-metal.md#administer-auth). ## Maintenance Stop the server before any maintenance command takes ownership of its root. Choose the procedure by the task: | Task | Guide | | --- | --- | | Check health or investigate disk usage | [Routine health checks](/docs/operations.md#routine-health-checks) | | Back up or recover repository data | [Backup and restore](/docs/migration-backup-restore.md) | | Move metadata independently of the database engine | [Metadata export/import](/docs/migration-backup-restore.md#scribe-server-export-metadata---repo-name---to-file) | | Upgrade a supported older schema | [Explicit migration](/docs/migration-backup-restore.md#scribe-server-migrate) | | Rebuild a lost or incomplete object index | [Locator rebuild](/docs/migration-backup-restore.md#scribe-server-rebuild-locator---repo-name) | All switches and aliases are in the [server CLI reference](/docs/reference/cli/scribe-server.md). ## Operating notes - Keep data, logs, and runtime directories separate; only the data directory needs backup, and it includes `auth-state.redb` when auth is enabled. That file also records the newest auth revision this server has seen: the server refuses auth answers from behind it (`auth_unavailable`, logged as `auth.rollback_detected`), so a copy of the auth data directory restored from an older backup cannot revive revoked credentials. Restore auth with [`scribe-auth restore`](/docs/reference/cli/scribe-auth.md#restore); see [rollback protection](/docs/auth.md#rollback-protection). - Keep the credential file and TLS key outside the data directory and its backups. - Run one server per data root; a second process fails on the lock. - Watch `submit.committed`, `submit.rejected`, `objects.published`, and `server.stopped` (which reports dropped diagnostic events) in the logs. Logs are never needed for recovery. For TOML-based local cluster startup, pepper provisioning, and idempotent user bootstrap, see [the cluster guide](/docs/cluster-usage.md). An optional [scribe-gateway](/docs/gateway-usage.md) can provide one client endpoint for several existing storage instances. Standalone operation remains supported.