Server setup and maintenance

For every command and option, see the server CLI reference. This guide covers configuration and service behavior. To set a server up, use scribe-ctl (tutorial 1, then add authentication) or run it by hand on bare metal or with Docker Compose.

All command-line tools support global --json; see JSON output for automation for results, errors, and exit codes.

Logging and JSON switches are listed in the CLI reference. See logging for filters, file retention, and timings.

Offline maintenance commands that open a data root take the same exclusive lock as serve, so they refuse to run beside a live server. Most select the root with --config FILE or --data-dir DIR; auth-init, repair-objects, recovery-archive, compact-objects, and recover-submissions accept only --data-dir DIR, and restore and restore-checkpoint take source and destination directories instead.

Two commands work beside a running server and take no data-root lock: checkpoint asks the server that owns --runtime-dir to take an online backup, and integrity-status reads saved verification reports from the runtime directory without opening the data root. Certificate and remote auth-administration commands do not need a data root.

init

init prepares a new deployment explicitly. For a cluster it manages, scribe-ctl setup runs it (first-server tutorial). By hand, init is followed by serve (bare metal, stage 1).

Creates an empty data root (descriptor, catalog, repos/, lock file) and optionally writes a configuration whose paths are relative to the configuration file. Data-root directories are created private (0700 on Unix). Omit --config-out when you only need the data root. Refuses a non-empty data directory, and refuses to overwrite an existing configuration without --force.

serve

Use serve for an initialized root.

SCRIBE_SERVER_CONFIG may name the configuration file; flags override it, and flag paths are relative to the current directory. Startup opens the descriptor, refuses unknown or newer formats, takes the lock, opens the catalog and every ready repository (discarding interrupted creations), logs every resolved path and limit, binds, writes <runtime-dir>/scribe-server.pid and <runtime-dir>/listen (the bound address, useful with port 0), then accepts connections.

Two modes:

• Plaintext TCP (no [auth] table and no --auth-*, --tls-* options). Anyone who can reach the port can act as anyone. The server refuses a non-loopback listen address unless --allow-non-loopback or allow_non_loopback = true is set, and logs server.plaintext_exposed when it is. A data root prepared with auth-init refuses to start this way.
• TLS with shared authentication. The four [auth] settings are required together, and the data root must have been prepared with auth-init. The socket then serves TLS 1.3 only, with the hybrid post-quantum key exchange X25519MLKEM768 and no other; listen reports tls://host:port. Clients authenticate with mutual TLS, presenting access certificates that auth issued; the server verifies them against the access CAs it fetches from auth and refreshes every five minutes, and until the first fetch succeeds it refuses every presented certificate. Every request is then authorized per repository. Setup is in bare metal, stage 2; the model in auth.md.

The runtime directory is created private (0700 on Unix). An existing runtime directory that group or other accounts can write is refused at startup (... is writable by other accounts (mode 777); restrict it with chmod go-w), because its files stop the server and request backups.

Stop with SIGINT, SIGTERM, or touch <runtime-dir>/shutdown-request. Admission stops and idle connections close. In-flight requests get up to drain_timeout_secs for network I/O; after that, sockets are interrupted and started storage jobs finish before exit. A stalled disk can extend shutdown.

After forced termination, acknowledged work remains durable under the storage assumptions. The next start cleans staging and unreferenced pack tails.

Configuration file

[server]
data_dir = "server-data"        # relative to this file
log_dir = "logs"                # omit for stderr
runtime_dir = "run"             # default <data_dir>/run
listen = "tcp://127.0.0.1:7447"
allow_non_loopback = false      # plaintext mode only; requires a trusted tunnel

[logging]
filter = "info"                 # or "warn,scribe_server=info,scribe_objects=debug"
format = "text"                 # or "json" (newline-delimited)
keep_files = 14                 # daily files retained in log_dir

[limits]
max_connections = 256
max_connections_per_principal = 64 # concurrent connections one user may hold
max_pending_per_source = 16     # connections one address may hold before its first request
storage_workers = 16            # blocking mutations/transfers; >=2
query_workers = 4               # reserved read queries; >0; storage+query+handshake <=1024
transfer_workers = 8            # >0 and <storage_workers; reserves control capacity
handshake_workers = 8           # concurrent authenticating handshakes; >0; never uses query/storage workers
staging_budget_mib = 512        # bytes admitted into quarantine across all connections
inactivity_timeout_secs = 300
handshake_timeout_secs = 10
drain_timeout_secs = 30
small_object_threshold = 16777236 # envelopes at or below this go to packs
pack_seal_mib = 256
max_workspaces_per_principal = 256 # workspace IDs one account may register per repository
max_untracked_locks = 4096      # locks on paths that do not exist at head, per repository
max_submit_entries = 100000     # change-set entries accepted in one submit; bounds publication gate hold
                                # advertised in server_info; `scribe import` batches a first population to it

[integrity]                     # optional; these are the defaults
enabled = true
mib_per_second = 1             # decoded verification payload budget
idle_secs = 1                  # yield to queued/running requests
interval_secs = 86400          # wait after each repository pass

[auth]                          # optional; all four keys together
namespace = "studio"            # registered with scribe-auth servers add --namespace
credential = "secrets/storage-a.credential" # scribe-auth credentials create --server
tls_cert = "pki/storage-a/server.pem"
tls_key = "pki/storage-a/server-key.pem"

scribe-server init --config-out writes [server] and [logging] with the values above, [limits] and [integrity] with every key commented out at its default, and [auth] entirely as a comment. Uncomment all four [auth] keys together: an [auth] table with no keys is refused (the [auth] table is empty; set namespace, credential, tls_cert and tls_key, or remove the table), and a partial one fails with auth requires namespace, credential, TLS certificate and TLS key.

[auth].credential is this server's credential file from scribe-auth credentials create --server. It carries the auth endpoint, the CA of the auth service's TLS certificate, the server's device key, and its credential certificate, so no separate endpoint or CA setting exists. The credential file and tls_key are secrets: both must be outside the data directory ([auth].credential PATH is inside the data directory; keep secrets outside it), because backups and host moves copy the data root. On Unix the credential file must grant nothing to group or other accounts, in a directory they cannot write; on Windows its access list must allow no other account more than reading its attributes, in a directory where no other account can delete entries. The server reads the credential file only at startup: restart it after replacing the file. See credential files.

Limits are validated rather than defaulted: a zero connection, pending, timeout, workspace, lock, or entry limit is refused with [limits].KEY must be at least 1, a MiB value that is zero or overflows with [limits].KEY must be a positive number of MiB that fits in 64 bits, and an unknown [logging].format with [logging].format must be text or json.

Admission is decided before any backend work:

• max_pending_per_source bounds the connections one address holds before it has completed its handshake and sent one complete request; an IPv6 source is counted per /64. When every slot is taken, the oldest pending connection is closed to admit a newcomer, so unauthenticated peers can delay a client by at most one handshake deadline. An idle connection counts as pending again until its next request.
• max_connections_per_principal bounds one user's concurrent connections, decided from the access certificate. Through a gateway the principal is the user it forwards, not the gateway. Beyond it a new connection is refused at its hello with busy (too many concurrent connections for this identity) and logged as conn.principal_limit. Anonymous connections have no principal and remain bounded by the anonymous limits.
• One user's request bodies in flight share a quarter of the listener's frame budget (32 MiB of 128 MiB), keyed by the same user id.
• New workspace registrations are limited to 64 per user per hour, separately from max_workspaces_per_principal (too many new workspaces this hour; reuse or remove existing workspaces and retry later, limit_exceeded).
• An anonymous chunk read of a public repository may make the server read at most 64 MiB of manifests to prove the chunks are public; a larger request is refused (a public chunk read may name at most 64 MiB of manifests at once; request fewer chunks).

The four worker counts shown are the defaults on a machine with 8 or fewer CPU cores. When a key is omitted, the default scales with the core count c (counted up to 64): storage_workers is max(2c, 16), query_workers is max(c/2, 4), transfer_workers is max(c, 8), and handshake_workers is max(c/4, 8). The small_object_threshold default is the 16 MiB chunk limit plus the 20-byte envelope header, so every chunk-sized envelope is packed. small_object_threshold and pack_seal_mib are recorded in a repository's object-store descriptor when the repository is created; changing them later affects only repositories created afterward.

For background verification, coverage reports, and restart behavior, see online integrity checking.

certs

Both service executables use the same offline certificate helper. See certificate rules, the CLI reference for options, and certificates and credentials for renewal and CA changes. Bare metal, stage 2 creates a complete certificate set.

auth-init, users, permissions

Follow adding authentication (or, by hand, bare metal, stage 2) before exposing an existing root to authenticated clients. auth-init prepares offline security ownership state; it does not create central users or issue service credentials. The ownership guide explains why plaintext working copies must be replaced with authenticated working copies.

The users and permissions conveniences talk to shared auth with an administrator credential file (--admin-credential). They use the same implementation as scribe-auth users and scribe-auth grants. auth-init does not register the server at auth: an auth administrator registers it (servers add) and issues the credential file named by [auth].credential (credentials create --server), as in bare metal, stage 2.

Maintenance

Stop the server before any maintenance command takes ownership of its root. Choose the procedure by the task:

All switches and aliases are in the server CLI reference.

Operating notes

• Keep data, logs, and runtime directories separate; only the data directory needs backup, and it includes auth-state.redb when auth is enabled. That file also records the newest auth revision this server has seen: the server refuses auth answers from behind it (auth_unavailable, logged as auth.rollback_detected), so a copy of the auth data directory restored from an older backup cannot revive revoked credentials. Restore auth with scribe-auth restore; see rollback protection.
• Keep the credential file and TLS key outside the data directory and its backups.
• Run one server per data root; a second process fails on the lock.
• Watch submit.committed, submit.rejected, objects.published, and server.stopped (which reports dropped diagnostic events) in the logs. Logs are never needed for recovery.

For TOML-based local cluster startup, pepper provisioning, and idempotent user bootstrap, see the cluster guide.

An optional scribe-gateway can provide one client endpoint for several existing storage instances. Standalone operation remains supported.