Background jobs

The web stack runs a dedicated worker service: its own program (apps/worker) and image, built on the same packages/service code as the API, with its own healthcheck, connection pools and secret boundary and no HTTP server. No browser or CLI process is needed while jobs run.

Durable scheduling and execution

SurrealDB's background_job table is the source of truth. A job has a stable ID, queue/realm, registered type, small identifier-only payload, next-run timestamp, unique generation, retry count, a generic last error and, while it retries, the last failure's safe detail (see Operations and health). Proposal creation and updates wake their repository job in the same database transaction as the proposal, snapshot and feed events. A failed transaction cannot publish work, and a process failure between database commit and dispatch cannot lose it.

The dispatcher reads at most 100 due jobs per second, using the queue/runAt index. Valkey uses atomic scripts, sorted sets and renewable ownership tokens to deduplicate queued/in-flight work and recover expired leases. All keys for a queue share one Valkey Cluster hash slot. Lease time uses Valkey's clock. Default leases last two minutes and renew every 30 seconds. Multiple worker replicas share this queue; concurrency defaults to two per replica and is capped at sixteen.

Delivery is at least once, not exactly once. A stale worker cannot acknowledge another worker's lease. Durable completion compares the job's unique generation, so an old completion cannot erase a newer wakeup, recreate/delete cycle or another completion. Handlers must still use CAS or durable operation IDs for side effects. Proposal handlers only write derived metadata through the existing proposal CAS; no native submit/merge is performed. SIGTERM stops admission and drains active work.

Valkey is disposable: due jobs remain in SurrealDB and are redispatched after cache loss. Retry delays grow from five seconds to five minutes with bounded jitter for handler failures. Per-proposal failures preserve progress across repository batches and retry after the pass. Failures never silently delete work. Jobs that need no further work return null and retire using the same generation check.

First consumer: proposal tracking

proposal.repository coalesces all open/draft proposals in one repository and storage incarnation into one recurring job. It queries up to 50 lightweight proposal references, reads native branch heads once per batch, and recalculates only stale comparisons. Additional pages are scheduled immediately for fairness; a completed pass schedules the next check after 30 seconds plus a small jitter. No files, chunks or complete histories are scanned when heads are unchanged.

This is bounded repository polling, not push delivery: storage currently has no change subscription. Submits through native clients are detected on the next scheduled pass without changing the CLI or wire protocol. Web proposal mutations wake the job immediately. A future native change-event adapter can wake the same job without changing handlers or queue semantics.

Startup performs a paginated, indexed scan of active web proposal metadata to ensure scheduled jobs exist, covering queue installation and recovery. It does not enumerate native repositories or read their histories. Closed/archived/merged proposals and uncertain in-progress merges stay pinned. Reopening wakes tracking. A storage incarnation mismatch retires the old-incarnation job.

Configuration and access

• WORKER_CONCURRENCY: 1–16 jobs per replica; default 2.
• PROPOSAL_CHECK_INTERVAL_SECONDS: 5–3600; default 30.
• WEB_SCRIBE_READER_CREDENTIAL_FILE: container path to the read-only credential file of the web services' service account, normally /etc/scribe/worker/reader.credential. This is the Compose name; Compose passes it to the worker, assets-index and assets-derive processes as SCRIBE_READER_CREDENTIAL_FILE, which is what a deployment outside Compose sets. Required when WEB_SCRIBE_AUTH_SERVER is set: all three refuse to start without it (SCRIBE_READER_CREDENTIAL_FILE is required for authenticated deployments). A credential also requires a tls:// storage endpoint. Plain/public-only deployments may omit it.
• Storage endpoint, TLS server name and trusted CA come from the same WEB_SCRIBE_* settings as the API. The auth endpoint and its CA come from the credential file itself; WEB_SCRIBE_AUTH_TLS_SERVER_NAME (passed as SCRIBE_AUTH_TLS_SERVER_NAME) names the identity auth's certificate carries when the endpoint's host is not it.
• Worker mounts only the database password and reader credential file; no API JWT key, auth administrator credential, account password or user browser sessions.

The worker, assets-index and assets-derive read as one service account (scribe-web by convention): an Auth account that has no password, never signs in, never administers and can hold at most writer. Its only credentials are credential files an auth administrator issues (see service accounts). A namespace reader grant lets it read every repository in the namespace, current and future, so a new private repository needs no extra step; to limit it to chosen repositories, grant it reader on each instead. The credential is read-only unless issued with --write, which the web services never need; it lives at most 365 days. The services reach auth at the endpoint the file records and fetch hour-long access certificates with its key, renewed at half their lifetime. Disabling the account revokes every credential it holds; browser logout does not.

• Create the account and issue its credential: issue the reader credential, which also covers the file's owner and mode on Linux (the services run as uid 1000). scribe-ctl --web does this itself.
• Replace it before it expires: service-account and web reader credentials. Every service rereads the file for each operation, so no restart is needed; a lapsed credential stops every job until a new one is in place.

Operations and health

docker compose up -d --build --wait api worker
docker compose logs --tail 50 worker
npm run test:data
npm run test:queue

The worker healthcheck requires a fresh local heartbeat after successful database dispatch and Valkey access. It does not claim every repository is reachable. Job records retain runAt, attempts, lastError (a generic line) and failure: when the attempt failed, the error's class, a stable code (an API error code, OpenSearch's HTTP status, the Scribe server's ErrorCode name or a system code such as ECONNREFUSED), a message only for errors whose text Scribe's services write (HttpError, OpenSearchError), and where in Scribe's code it was thrown; a successful run clears it. The worker also keeps each job's history in background_job_event, the newest 50 events per job: every failed attempt, a run that asked to be retried, the first success after failures, and runs whose handler counted what they got through (the asset processing jobs count files processed, failed and crashed); idle runs are not recorded. A history write that fails is logged as job_history_failed and never fails the job. job_retry, job_completed, queue_unavailable and valkey_error logs contain job IDs/types, sanitized status and a diagnostic (the error's class, symbolic code and where in Scribe's code it was thrown), never an error message, a secret or an upstream payload. A Valkey connection that keeps failing is logged at most once a minute, with a count of the failures in between. Authorization failures of the worker account carry one of two codes, which the repository's Settings → Asset index section explains, with the fix, to Maintainers: worker_repository_hidden (the account has no reader grant on the repository its message names or on its whole namespace, or the repository is gone; the section shows the scribe-auth namespace-readers set command for the repository's namespace) and worker_credential_refused (the credential expired, was revoked or its account is disabled; the section shows the scribe-auth credentials create --service command that issues a new one). Missing worker access cannot expose private content through REST, MCP or feeds: those independently check the requesting user's current access.

Orphaned data

Two worker jobs find and remove what the web tier still keeps about repositories that are gone. An Auth administrator starts both from /admin/leftovers; the API only wakes them and reads their payloads.

• maintenance.leftovers (one per realm) counts every store and walks the complete repository directory as the service account. It then parks: its runAt is never due. Its payload holds the report until the next scan.
• maintenance.purge (one per realm and repository) names its repository target, not repo, so the jobs store it empties never includes the purge itself. A run removes at most 25 batches. The job reschedules itself at once while more remains, then parks with its outcome and the removed counts. A repository whose stores keep refilling fails the job after 20 passes, and it retries with the usual backoff.

Asset jobs of a deleted repository fail every run with worker_repository_hidden and retry forever. The purge removes them first.

Add another job type

1. Define a small payload schema and a stable deduplication ID for the resource.
2. Register its durable schedule through the JobRepository data interface. When scheduling follows another durable mutation, include wakeJobSql in that data adapter's transaction; do not perform an unprotected DB-write/queue-publish pair.
3. Implement a JobHandler(job, signal) returning the next runAt/payload, or null when complete. Bound work per invocation, check cancellation between operations, and make side effects idempotent. Throw to use the common retry policy.
4. Register the handler by type in apps/worker/src/main.ts. Keep privileged credentials scoped to that handler's adapter. The generic worker and queue know nothing about proposal semantics.
5. Add invariant tests and the relevant persistent adapter contract. The existing queue contract tests concurrent enqueue/claim, recovery and stale-owner fencing.

The Valkey adapter uses the existing Redis client dependency and standard Valkey atomic scripts; it adds no broker or queue package. The database and transport contracts allow future adapters.