On-disk format specification

Normative description of every byte Scribe writes to disk. Version 1 of each component is frozen; a change means a new version number. All multi-byte integers are little-endian unless a field is explicitly marked ordered (big-endian, used only inside lexicographically compared index keys). [n] means n raw bytes. str16 is a u16 byte length followed by UTF-8; str32/bytes32 use a u32 length. Digests are BLAKE3-256, 32 bytes, never truncated.

Reference implementation: crates/scribe-format (encodings, keys, descriptor), crates/scribe-objects (pack and loose layout), crates/scribe-metadata (tables), crates/scribe-workspace (workspace records).

1. Identifiers and hashing

Hash algorithm identifier 1 = BLAKE3-256 appears in every envelope. Hex renderings are lowercase.

2. Format descriptor file SCRIBE_FORMAT

Present at the root of a data root, a repository directory, an object store, and a workspace .scribe/. UTF-8, LF or CRLF, one key = value per line, keys [a-z0-9-]+, values without newlines. The first line is exactly scribe-descriptor = 1; a kind line is mandatory. Unknown keys are kept. A required-features value is a comma-separated list; an opener that does not recognize every listed feature refuses to open for mutation.

Openers compare each version they know against the descriptor before opening any database; a mismatch is a refusal that names scribe-server migrate.

3. Object envelope v1

Every stored object, loose or packed, is an envelope:

Identity is the digest of the original canonical payload, after decompression. Codec 0 stores those bytes directly. Codec 1 stores u64 original length followed by independent blocks: u8 codec | u32 original length | u32 stored length | stored bytes, all integers little endian. Each block contains 1 to 262144 original bytes. Block codec 0 is raw (equal lengths); codec 1 is a standard LZ4 block with strictly fewer stored bytes. Unknown codecs, invalid lengths, truncation, and trailing bytes are rejected. There is no terminator; the original object length determines the end.

Repository creation fixes the storage policy for its object store. It cannot be changed through the API or CLI. Under lz4, an object retains codec 0 if the complete encoded representation, including the original-length prefix and block headers, is not smaller. Locations and compression are never part of object identity. Manifests continue to describe original chunk bytes.

4. Payloads

4.1 Manifest v1

u16 version = 1
u32 flags = 0                (unknown bits reject)
u8  content codec = 0        (logical chunk contents are original bytes)
u16 chunking profile         (1 = FastCDC-2020, level 1, seed 0, 1/4/16 MiB)
u64 total size
[32] file content id
u32 chunk count
repeat chunk count: [32] chunk id, u32 chunk length

Rules: sum(length) == total size; chunk count == 0 ⇔ total size == 0; 1 ≤ length ≤ 16 MiB; empty file content id is blake3("") (af1349b9…f3262); repeated chunk ids allowed; total size ≤ 1 PiB; chunk count ≤ 2^30. Executable bits are not in the manifest. Files no larger than the profile minimum (1 MiB) are exactly one chunk.

4.2 Change set v1

u16 version = 1
u32 entry count            (≤ 4,000,000)
repeat, strictly ascending by path bytes, no duplicates:
  str16 canonical path
  u8  action: 1 add, 2 edit, 3 delete
  u64 expected base change   (0 for add; the change that produced the base state otherwise)
  if action ≠ 3: [32] manifest id, [32] file content id, u64 size, u8 mode (0 regular, 1 executable)

4.3 Commit v1

u16 version = 1
[16] repository id
[16] branch id
u64 change number            (≥ 1)
u64 parent change            (0 = none; must be < change)
u8  branched-from present    (0 or 1)
  if 1: [16] source branch id, u64 source change
[32] change set id
[16] operation id
[16] workspace id
u64 timestamp, Unix milliseconds UTC
str16 author                 (≤ 256 bytes)
str32 message                (≤ 65,536 bytes)

5. Canonical paths

UTF-8, / separator, relative, no empty/./.. components, ≤ 8192 bytes, components ≤ 255 bytes, no control characters, none of < > : " | ? *, no reserved device name stem (CON PRN AUX NUL COM1-9 LPT1-9, any case, with or without extension), no component ending in . or space, no component equal to .scribe (any case). Collision key = NFC normalization then Unicode lowercase; two live paths with equal keys cannot coexist on one branch.

6. Object store layout

<store>/SCRIBE_FORMAT
<store>/locator.redb
<store>/loose/<hh>/<kk><digest hex>      hh = first digest byte in hex, kk = kind in 2-digit hex
<store>/packs/<pack id, 8 hex digits>.pack
<store>/staging/                        quarantine temporaries (same filesystem)

Objects whose envelope length ≤ small-object-threshold (default 262,144) are packed; larger ones are loose files containing exactly one envelope.

6.1 Pack file

header (16 bytes): "SCPK" | u16 version = 1 | u32 pack id | u16 reserved = 0 | u32 flags = 0
entry (repeated): "SCPE" | [33] object key | u64 envelope length | envelope bytes

Entry length = 45 + envelope length. Packs are append-only. Every pack except the highest-numbered is sealed. The open pack rolls over when an append would exceed pack-seal-bytes (default 256 MiB). After a crash the open pack may have bytes past the last committed length; open truncates them (they were never referenced). A pack shorter than its committed length is corruption and refuses to open.

6.2 Locator database locator.redb

Tables: locator key [u8; 33] object key → value:

u8 version = 1
u8 location: 1 loose, 2 packed
  loose:  u64 envelope length
  packed: u32 pack id, u64 entry offset, u64 entry length

meta (&str → bytes): open-pack (u32 LE pack id), open-pack-committed-len (u64 LE). The locator is an index: ObjectStore::rebuild_locator recreates it from packs and loose files (scribe-server verify reports; rebuild is offline).

6.3 Publication order of a batch

1. staged file data flushed at staging time (loose candidates);
2. loose renames into shard directories / pack appends;
3. pack file flushed once per touched pack;
4. each touched shard directory flushed once; loose/ flushed if a shard was created;
5. locator transaction commits (Durability::Immediate).

Objects are never deleted, replaced, or compacted in v1.

7. Data root layout

<data>/SCRIBE_FORMAT
<data>/LOCK                    exclusive OS lock (flock / LockFileEx) held by any owning process
<data>/catalog.redb
<data>/auth-state.redb          workspace and operation ownership, schema 1 (only after auth-init)
<data>/repos/<repository id hex>/SCRIBE_FORMAT
<data>/repos/<repository id hex>/metadata.redb
<data>/repos/<repository id hex>/objects/            object store (section 6)
<data>/repos/<repository id hex>/metadata.redb.gen1  previous generation after a migration (optional)
<data>/repos/<repository id hex>/MIGRATION, MIGRATED migration checkpoint / record (optional)

Logs and runtime files (scribe-server.pid, listen, shutdown-request) live in separately configured directories and are disposable.

7.1 Catalog catalog.redb (schema 1)

Creation writes creating, builds the directory, then ready; startup removes creating leftovers with their directories.

8. Repository metadata metadata.redb

Schema 3 is what this build serves. It consists of the schema-1 tables below, the schema-2 branch tables, and the schema-3 collision index (section 12). Every value starts with a u8 record version (1). Keys:

Ordering guarantees: paths never contain 0x00, so path ‖ 0x00 ‖ change keeps one path's revisions contiguous and before path ‖ '/' ‖ …. Branch prefixes make per-branch scans key-prefix scans. current_fold is rebuildable from current; commits, history, change_paths are re-derivable from the immutable commit and change set objects (slow path); everything else is authoritative and must be backed up.

Operation fingerprint = blake3("scribe:operation-fingerprint:v1\0" ‖ repo ‖ branch ‖ workspace ‖ change set id ‖ u64 len ‖ author ‖ u64 len ‖ message).

9. Workspace .scribe/ (schema 1; schema 2 adds workflows_v2, section 12)

.scribe/SCRIBE_FORMAT
.scribe/state.redb
.scribe/ignore              optional workspace-local ignore rules (not versioned)
.scribe/cache/              object store (section 6) holding objects ≤ 256 KiB and manifests
.scribe/tmp/                reconstruction temporaries (`<hex>-<index>-<pid>.part`), cleaned at open
.scribe/recovery/           displaced local contents: `<unix ms>-<32 hex random>`; the sync report maps each to its path
.scribe/LOCK                local operation lock

state.redb tables:

10. Backup and export containers

BACKUP_MANIFEST: text, first line scribe-backup-manifest = 1, then key = value lines (created-ms, source-incarnation, repository-<id> = <name> head=<n> commit=<hex>) and one file = <blake3 hex> <size> <relative path> per copied file.

Metadata export (.scmx): "SCMX" | u16 version = 1 | [16] repository id | u32 table count | per table: str16 name, u64 row count, (bytes32 key, bytes32 value)* | [32] blake3 of everything before it.

11. Limits that are part of the format

Path 8192 B; component 255 B; file 1 PiB; chunk 16 MiB; manifest chunks 2^30; change set entries 4,000,000; author 256 B; message 64 KiB; repository name 128 B; view prefixes 256.

12. Branch extension: repository schema 2 and 3, workspace schema 2, commit v2

The v1 encodings above are unchanged. Repository schema 2 adds:

Integers use little endian. Branch rows retain their v1 byte layout under schema 2; state 2 means an unpublished snapshot. Current/history/collision keys retain the branch reservation. Snapshot history rows anchor the inherited file state; user-facing path history follows the recorded source origin.

Commit v2 is u16(2), a u32-length-prefixed complete v1 commit payload, then a u8 optional-parent flag. A present parent is BranchId plus u64 ChangeNumber. Its identity hashes scribe:commit:v2\0 followed by this complete payload. The stored commit index row retains its v1 layout and contains the v2 identity. Normal commits still use commit v1. The repository's global change sequence orders all commits; branch heads and first/second parents describe ancestry.

Workspace schema 2 adds workflows_v2 (UTF-8 string keys to explicit versioned bytes). It holds pending branch/switch intent, one active resolution workflow, and per-path immutable base/ours/theirs/target references plus selection and completion flags. Planning completes before application starts. File application, base/opened updates, and the resolution completion marker are coordinated with recovery protection. Accepted-submit finalization clears the active workflow in the same local transaction as its pending submit and baseline changes. See scribe-workspace/src/workflows.rs and scribe-client/src/conflicts.rs.

Repository schema 3: indexed historical path relations

Schema 3 includes all schema-2 tables and adds fold_history_v3. The key is BranchId(16) ‖ UTF-8 collision_key(path) ‖ 0x00 ‖ ordered_u64(change); only the ordered change segment uses big endian. The value is bytes_u16(original canonical path) ‖ HistoryRecord(v1). The table name and repository schema version identify the encoding; v1 bytes are unchanged.

At one folded path/revision a live addition wins over a deletion (case-only replacement). Branch snapshot pages populate the inherited anchors. Every subsequent acceptance updates this index in the same durable transaction as history and branch head. Exact and ancestor states use a reverse bounded range seek; descendants page over distinct folded paths, counting tombstones against the page budget. Reads at pinned revisions are stable across later submissions.

migrate --branches builds the index in 1,024-row Immediate transactions while holding exclusive root ownership. The schema marker advances only after the whole index is durable. Replaying after interruption is idempotent. Doctor checks both directions against history without mutation. Physical backups and logical export/import include the table.