Architecture

Scribe is a centralized version-control system for large game projects. The server is authoritative; clients keep working copies and lightweight local metadata under .scribe/_state/. .scribe/ itself is Scribe's namespace inside the repository: only _state/ beneath it is reserved local workspace state; everything else under it is ordinary versioned content. There is no distributed history.

Repository layout

crates/ contains Rust libraries and executables. apps/ contains applications: apps/web/ owns the complete web workspace, and native clients belong alongside it (apps/macos/ is the SwiftUI shell, generated with XcodeGen; apps/windows/ is the WinUI 3 shell, built with the .NET SDK). crates/scribe-ffi is the shared C ABI those native clients link; it is not macOS-specific. integrations/ contains host integrations such as the VS Code/Cursor extension in integrations/vscode/. Each application or integration owns its build configuration and tests.

tools/ contains repository checks, generators, scenarios, and performance tools. scripts/ contains user-facing installers; docker/ contains core service deployment; docs/ contains shared documentation. The Rust workspace stays at the repository root.

Crates and boundaries

crates/
  scribe-core          ids, digests, canonical paths, change numbers, error codes            (no I/O)
  scribe-format        canonical byte encodings, version dispatch, keys, descriptor          (no I/O)
  scribe-platform      every OS-specific behaviour: flush primitives, positional and zero-copy I/O,
                       cache hints, preallocation, reflink clones, rooted directory handles, free space;
                       one module per OS; the only crate with OS unsafe
  scribe-api           transport-neutral contract, shared repository config and lock-rule validation
  scribe-objects       chunking profile, staging, loose + pack object store, locator, verification
  scribe-metadata      redb catalog and per-repository tables, the acceptance transaction
  scribe-workspace     .scribe/_state/ state, ignore rules, scanner, journals, branch/merge workflows
  scribe-server        Service: repository operations, publication sequence, budgets, authorization
  scribe-protocol-tcp  TCP v2 framing and codecs (LE production, BE test), protocol-4 caller hello,
                       client + server adapters
  scribe-tls           blocking client and async server rustls TLS 1.3 streams (aws-lc-rs, X25519MLKEM768),
                       device keys, credential files, client-certificate verification and identities
  scribe-auth          auth wire protocol (SAU2), client and client sessions; service (with the client
                       CAs) and admin CLI behind features
  scribe-cert          CA and certificate generation through rcgen (server PKI and auth's client CAs);
                       verification through scribe-tls
  scribe-client        workflows: add/edit/delete/move/revert/status/reconcile/submit/sync/export/get/log/locks/branches/merge
  scribe-ffi           C ABI for native GUI clients; workspace lock is per call, not per window;
                       unsafe only for FFI trampolines, pointer ownership, and C string conversion
  scribe-log           logging configuration, sinks, terminal detection for executables
  scribe-server-cli    the scribe-server executable
  scribe-cli           the scribe executable
  scribe-auth-cli      the scribe-auth executable
  scribe-runner        the scribe-runner executable: flow jobs for the web API, client compiled in
  scribe-ctl           the scribe-ctl executable: cluster setup (definition, plan, journal) and process supervision
  scribe-gateway       the scribe-gateway executable: repository routing, placement, and backend
                       inventory; its configuration library is shared with scribe-ctl
tools/
  scribe-perf          direct measurement helper for JS scenarios; never shipped
  scenario.js          ES-module entry point for reusable user-workflow plans
  scenarios.json       services, datasets, compression variants, and scale
  scenarios/           service/client objects, dataset factories, plans, and tests

Dependency direction (arrows point at what is depended on):

scribe-cli        -> scribe-client -> scribe-protocol-tcp (client) -> scribe-api -> scribe-format -> scribe-core
                                   -> scribe-workspace -> scribe-objects -> scribe-platform
                                   -> scribe-tls, scribe-auth (client only)
scribe-ffi        -> scribe-client, scribe-auth (session), scribe-tls (saved trust)
apps/macos        -> scribe-ffi C ABI (Swift does not speak the wire protocol)
apps/windows      -> scribe-ffi C ABI (C# does not speak the wire protocol)
scribe-server-cli -> scribe-protocol-tcp (+server) -> scribe-server -> scribe-metadata, scribe-objects
                  -> scribe-auth (+admin), scribe-cert, scribe-tls
scribe-auth-cli   -> scribe-auth (+server, +admin), scribe-cert, scribe-tls
scribe-runner     -> scribe-client, ureq (web API), scribe-platform (process trees, service entry)
scribe-gateway    -> scribe-protocol-tcp (+server) -> scribe-server -> scribe-metadata
                  -> scribe-tls, scribe-auth (client only), scribe-platform
scribe-ctl        -> scribe-gateway (configuration), scribe-protocol-tcp (client inventory)
                  -> scribe-auth (+admin), scribe-cert, scribe-tls, scribe-platform

Rules the crate graph enforces:

• scribe-core and scribe-format have no networking, no database, no filesystem beyond OS randomness.
• scribe-server never depends on a wire adapter. Adapters depend on it. Authorization is enforced inside scribe-server through the AuthProvider contract; the executable constructs the provider.
• scribe-cli links no server code and no auth database: the TCP crate's server module is behind the server feature and the auth service behind scribe-auth/server, which only the service executables enable (cargo tree -p scribe-cli -e normal,build --locked).
• Client workflows program against scribe_api::RemoteOps. The in-process adapter (scribe_server::LocalRemote), TCP with the production codec, and TCP with the big-endian test codec run the same contract tests.
• Libraries return typed results and emit tracing events. Only executables install a subscriber and write to stdout/stderr.

Data flow of a submit

client                                  server
------                                  ------
analyze files: chunk + hash (profile 1)
persist upload intent, pin object ids --> durable protection until intent finalization
presence(chunk ids)  ----------------->  locator lookup
put_objects(chunks, manifests,          stage to quarantine, verify digests,
            change set) in one batch --> validate structure, publish batch:
                                         pack append / loose rename, flush files,
                                         flush directories, locator commit
persist candidate + OperationId
submit(op id, change set id) ----------> candidate number = head + 1; policy and
                                         config checks; prepare snapshot pages; publish envelope
                                         (durable), all before the gate
                                         gate: confirm heads unchanged, else re-prepare
                                         accept(): one redb transaction revalidates
                                         bases + locks + collisions, publishes staged roots,
                                         commits/operations and upload cleanup marker, head++
                                         archive committed metadata outbox and publish durable HEAD
                                    <--- acknowledge only after commit and archive barrier
finalize baselines, drop candidate

Small snapshot and history plans prepare concurrently in bounded memory; larger plans stage in bounded transactions under a per-repository FIFO submit turn. Compatible small submissions share bounded group publication. The publication gate covers final acceptance and its archive barrier; uploads, asset hashing, verification, and authorization checks happen outside it. Archive I/O happens after the metadata write transaction closes.

The accepted .scribe/config.toml on main supplies lock policy for every branch. Policy object reads and compilation happen before publication; the server rechecks the policy revision under the gate and enforces required locks at acceptance. Changing the decoded ordered lock rules on main requires a Maintainer in authenticated deployments. Profile-only and formatting changes remain Writer operations. Authorization compares the requested replacement with its exact accepted base; acceptance rechecks that base under the publication gate. A client's local rules or reconciliation marker cannot bypass server validation.

Explicit moves carry exact predecessor revisions, independently of content identity. History follows indexed predecessor and successor edges through moves, forks, and merges. Cross-branch queries pin a repository snapshot. Merge origin queries follow predecessors toward the pinned base, permit forward links only within that base's ancestry, and reject ambiguity. Rename planning never infers identity from equal bytes or scans a complete tree.

Client state records pending move dependencies, offline edit intent, and grouped resolution choices durably. Resume and abort use those records and preserve displaced files in recovery. reconcile checks current server revisions and locks without replacing working files or changing their baselines.

Status exposes one typed row per physical path, combining opened work, local inspection, moves, resolution, and deferred sync. Remote enrichment is optional and published only after all required reads succeed; missing remote data is unknown. Historical get uses verified reconstruction and exclusive publication to an explicit destination without changing workspace metadata or its cache.

Data flow of a sync

head(branch) -> verify persisted branch/incarnation anchor and envelope ancestry -> plan:
   fresh / view changed / explicitly re-anchored: verified live pages as-of target merged with baselines
   otherwise: verified changed-range pages between synced and target, in either direction
              one latest event per selected path, ordered by path;
              snapshot summaries prune unchanged and unrelated subtrees;
              include local submit and interrupted-target revision frontiers
journal (redb) holds the verified plan and target anchor -> apply in batches of <=256 files / 512 MiB:
   opened paths deferred, existing local files displaced to .scribe/_state/recovery,
   chunks from cache or fetched in batches, temp file per target in .scribe/_state/tmp,
   full-content verification, exclusive rename into place through rooted directory handles,
   one best-effort filesystem flush per batch where supported (Linux)
watermark advances only when every entry is done; failures leave the journal for the next run

Workspace selection changes

Workspace profiles and view changes remain client workflows over RemoteOps. Accepted .scribe/config.toml contents are read at a pinned branch revision and verified through state proofs and their manifest and chunk digests. The client saves the resolved selection in a durable view-change record, then plans included paths and local baseline exclusions through the sync journal. Exact-file inclusions combine an exact state query with ordinary directory tree pages.

The prior selection remains active during application. Unchanged selected files retain local edits; opened exclusions defer completion, and displaced contents remain in recovery. After all paths complete, the client repeats the existing workspace registration with the new prefixes. One local immediate transaction then publishes selection and watermark together and clears the intent/journal. A lost registration reply keeps the intent for retry. See workspace views.

scribe-api::repository_config owns the shared TOML parser and typed sections; scribe-workspace uses its selection validation. Both clients and the service parse the complete document. Profiles use the selected branch; active lock rules always use accepted main. Config validation and section authorization add no object/index layout changes. Lock-policy snapshots carry complete TOML through the existing bounded text field. See repository configuration.

Repository composition and integration

Versioned [mounts.NAME] entries in .scribe/config.toml identify child repositories, branches, and accepted revisions. Sync preflights the selected composition, then applies each workspace through its own journal. Every child operation authorizes independently. Completed parent files can advance their own watermark while a separate mount marker keeps the overall composition incomplete and retryable. See repository mounts.

Project publication is client orchestration over ordinary per-repository submits and outcome queries. Its root journal persists participant operations before transmission; children become accepted before the parent pins those revisions. There is no cross-repository acceptance transaction. See project workflows.

Merge plans cover the complete source delta. A partial workspace materializes included results and retains clean excluded results as immutable decisions in the same resolution journal. Submission reuses their accepted manifests, and finalization creates local baselines only for included paths. Excluded conflicts must be included before resolution; they cannot disappear from coverage because of a view filter. Cherry-pick shares the resolver but records source provenance instead of a merge parent. See partial-view integration and cherry-pick.

Features and dependencies

Third-party crate versions, enabled features, licences, and replacement constraints are documented in dependencies.md.

Shared authentication

scribe-auth-cli supplies the scribe-auth executable. Its scribe-auth library separates the binary TLS auth client from feature-gated database and service code. scribe-tls wraps blocking rustls TLS 1.3 client I/O and Tokio/rustls server I/O for auth, storage, and gateway connections. The AuthProvider contract lives in scribe-api, so scribe-server::auth::AuthorizedRemote enforces repository roles, workspace ownership, and operation ownership without depending on an auth wire adapter. The storage executable constructs and injects the provider at startup.

Identities. Every credential holder (a signed-in person, a web session, a service account, a runner, a storage server, a gateway, an administrator) generates its own ECDSA P-256 device key, and the private key never leaves it. Auth records only the key id, the BLAKE3 digest of the public key's SubjectPublicKeyInfo, and registers each public key at most once. A certificate names its holder by exactly one SPIFFE URI, spiffe://<trust domain>/user/<user id>/credential/<credential id> or spiffe://<trust domain>/server/<server id>/credential/<credential id>, never by a username. See identity, device keys, and certificates.

Two client CAs. Auth keeps two private CAs in its database, created by scribe-auth init or migrate. The credential CA signs a credential certificate that lives as long as the credential (a password session 90 days by default) and is accepted only by auth. The access CA signs an access certificate for the same key that lives one hour and is accepted only by storage servers and gateways; issuing one writes nothing at auth. Clients renew access certificates at half their lifetime measured by local elapsed time, so wall-clock drift does not interrupt work. These CAs are separate from the operator's server PKI (certs init-ca, certs issue), which still certifies the servers themselves.

Registry check. A certificate alone authorizes nothing. When a storage server accepts a connection it verifies the access certificate against auth's access CAs (fetched with the SAU2 trust-bundle verb and refreshed every five minutes), then asks auth with its own credential certificate whether the credential is live, the certificate's key is the registered key, and the account is enabled. It caches the answer for the authorization lease of at most 30 seconds, so revoking a credential or disabling an account takes effect within that time regardless of certificate lifetime, and a leaked CA key alone certifies nothing usable.

Delegation through gateways. A gateway is a service account holding a namespace-reader grant (for inventory) and a delegate grant for each namespace it fronts. It verifies each client's access certificate, then connects to storage with its own access certificate and forwards the user's certificate in the hello (caller 2, delegated). Storage verifies the forwarded certificate itself and auth confirms both credentials and the delegate grant. The principal is the user, so per-principal limits and authorization apply to the user, and a compromised gateway can act only for users whose access certificates it currently holds, only in its namespaces. See gateways and delegation.

Rollback protection. Every applied administrative change and every logout advances auth's revision in the same transaction; every SAU2 reply names the auth incarnation and revision. Auth keeps a revision witness file outside its data directory and its backups and refuses to start from a database behind it. Each storage server durably records the newest incarnation and revision it has seen in auth-state.redb before acting on a newer answer, and refuses answers from a lower revision of the same incarnation. scribe-auth restore is the one way back from a backup: a new incarnation with every earlier credential disabled. See rollback protection.

Storage TLS sessions use protocol 4; unauthenticated TCP v2 retains its encodings. Security ownership lives in an explicitly initialized, separately versioned auth-state.redb, with a root required-feature marker that prevents older executables from opening secured data. This state is included in normal storage backups. See auth.md and the wire format.

Connection scheduling and durable storage

Each server owns a Tokio runtime. Storage has up to four I/O workers (bounded by available parallelism); auth has two. No thread is assigned to an idle connection, incomplete TLS handshake, or incomplete metadata frame. Storage connection buffers are 64 KiB each for reading and writing. Frame length validation and canonical codecs are shared with the blocking clients. One request at a time owns a connection, including any buffered read-ahead from a streamed upload.

The synchronous service API runs on blocking workers admitted through a semaphore, acquired before spawning. Storage defaults to 16 mutation/transfer jobs, at most 8 active upload/download jobs, and 4 additional reserved read- query jobs. Transfer permits are acquired first, leaving mutation capacity; read queries have their own admission permits so publication and metadata-writer waits cannot consume all query workers. Handshake authentication, which may wait on the auth service or a gateway backend, runs on its own pool of 8 handshake workers by default, so unauthenticated peers occupy neither query nor storage capacity. The runtime blocking-thread cap is the sum of storage, query, and handshake workers. Auth defaults to 16 storage jobs and sends responses asynchronously after each job finishes. The default connection caps are 256 for storage and 128 for auth and are configurable independently of the workers. Queued requests apply backpressure at the connection; there is no unbounded stream of jobs submitted to Tokio's blocking queue.

Active storage transfers occupy a bounded blocking worker while the synchronous streaming API consumes/produces bytes. Their socket I/O is driven by Tokio through the synchronous bridge. Plaintext file-range downloads retain the platform zero-copy path: the socket is deregistered, used solely by the transfer worker, then re-registered as nonblocking. Objects smaller than 64 KiB share buffered writes to avoid a flush and socket registration per tiny object. TLS always passes through rustls and uses buffered writes.

Shutdown closes idle connections and stops admitting queued requests. Started jobs are awaited. At the drain deadline, network I/O is cancelled (including zero-copy writes), but a started storage commit is never abandoned or acknowledged early. A stalled disk can therefore extend shutdown beyond the drain deadline while the process retains its data-root lock.

Tree-page reads validate the branch head and select current or historical rows within one redb read transaction. Reads proceed without taking the publication gate while each page stays consistent with the requested head. Reserved query workers remain useful while mutations wait for durable publication. Concurrency tests verify reads while publication is blocked and snapshot consistency across later accepted changes.

Storage uses Durability::Immediate and publishes objects, then the commit envelope, then metadata, then the independent metadata archive before sending a response. Writable-handle reads clone the latest independently archived redb snapshot under a short mutex; archive I/O never holds that mutex. During a slow archive flush they keep serving the previous published state. A failed archive fences mutations until restart while that published state remains readable. Async networking bounds connection overhead and blocking work; filesystem flush latency and contention between writers still affect request latency.

Bounded publication and object lifecycle

A FIFO submit turn orders publication per repository. Small submissions prepare read-only metadata plans concurrently, retaining at most 512 data rows / 4 MiB per plan. Pending snapshot bytes are shared by reference. Plans bind repository, incarnation, heads, lock epoch, and snapshot allocation; stale plans are rebuilt. Larger submissions use the same row builder with bounded durable staging and an undo inventory under the turn. Readers see only accepted revisions.

Ordinary buffered submissions may share a group of up to eight requests, with at most 512 data rows / 4 MiB across the entire group. The leader publishes all individual envelopes durably, then one Immediate transaction accepts the rows, ordered ledger entries, roots, and operation outcomes. One archive publication completes before any group member is acknowledged. A stale member aborts the whole transaction. The queue holds at most 64 requests and coalesces arrivals with a requested wait of 0.5 ms. Request threads hand leadership to the next batch; no background commit is cancelled when a connection disappears.

Policy changes, integrations, moves, oversized plans, and requests that cannot join a valid bounded group use individual publication. Streamed rows remain invisible until their constant-row final acceptance. Buffered acceptance writes its bounded rows directly with the ledger, so it needs no pending marker or preparation archive. The existing journal represents groups without new bytes.

Lock acquisition, release, and workspace removal join the same submit turn before taking the publication gate. Workspace removal settles accepted lock updates outside that gate, keeping its owner-index check bounded. This preserves the prepared lock epoch and reserved rename generations. FIFO turn admission prevents new contenders from overtaking queued turns; each group is bounded. Lock mutations can wait for the current preparation; reads remain available. Neither snapshot construction nor object flushing moves into the publication gate.

Clients persist workspace upload pins before transfer or reuse. Pins retain unfinished per-intent uploads. Finalized submit outcomes release only their own pins. Their restartable cleanup runs after releasing the submit turn and handing off group leadership. Unresolved intents remain until retirement or explicit offline abandonment. Offline collection traces accepted history plus active pins using disk-backed indexes, verifies retained bytes into new packs, and installs the store through a durable, recoverable swap. The incremental mode marks the same roots once per metadata boundary, reuses unchanged payloads with hard links, and rewrites fragmented packs within an operator budget. It remains offline and needs repository-sized mark and locator indexes, but does not copy all retained payload bytes. Content identity remains independent of storage location. See storage records and server commands.

Portable state and recovery retention

Each local radix link carries full content and revision digests. ContentRoot identifies the live path/content/mode map independently of edit history and numeric page allocation. RevisionRoot additionally binds manifests, last-change numbers, deletion tombstones, and summaries. V5 commit envelopes bind both. The proof API and shared client verifier can establish sparse-page completeness against trusted accepted roots. Logical proofs continue through bounded subtree fragments; the client retains the authenticated frontier and determines completion. Valid deep or wide trees remain synchronizable within the per-response limits. Normal workspace sync now verifies live pages, changed ranges, point states, configuration and omissions before applying its journal. Each branch keeps a trusted commit identity separately from its applied revision. Head checks follow authenticated parent identities to that anchor; backward sync never lowers it. First contact trusts the connected authority. An incarnation change or rollback is refused until an explicit trust decision. Checks checkpoint after 4,096 envelopes so retries can continue bounded work. See workspace trust.

Metadata recovery combines the transactional outbox with independently placed, hash-linked archives and consolidated logical bases. Offline retention holds the root lock, verifies a new base, prunes covered journal rows/segments, and can reclaim unreachable snapshot pages. Historical state and retry outcomes remain. This controls redundant and temporary storage; total accepted history still grows. The extra digests and durable archive writes have storage, hashing, and flush costs. See the canonical formats and operator procedures.

Submit scheduling

Publication takes one repository-wide FIFO submit turn. Small metadata plans prepare concurrently before that turn and compatible requests share a bounded commit group. Larger plans stage under the turn, so an import or another large change can still delay a small submit, even on a different branch. Uploads happen outside that turn, and reads use archived published snapshots. Work scales with the changed paths and their metadata, not just uploaded file bytes.

For the first release, large-submit queueing is an accepted tradeoff for bounded memory and simpler durable recovery. Parallel durable staging of large requests is not implemented. Evaluate mixed large/small submit latency on the intended server storage before release; there is no measured daily-submit latency guarantee yet.

Future distribution

A repository remains on one authoritative server. Buffered plans bind logical repository/incarnation and parent state, while physical snapshot page IDs stay outside portable state commitments and object identity. Storage placement is separate from chunk, manifest, and changeset identity; client operations continue to use RemoteOps, and the service does not depend on a wire adapter. These boundaries permit future remote preparation or distributed object storage without embedding machine addresses in content identities. There is no distributed writer, failover protocol, or unused compatibility layer in this implementation; future distribution still needs an explicit protocol and recovery design.