Tests & qualification

Scribe's tests check that accepted history survives failures, local work is protected during sync, and clients behave consistently across supported platforms. This guide explains how to run the checks from a source checkout and what their results can tell you.

Save reports

Both JS entry points accept repeatable --report=FILE (or --report FILE). The filename extension selects JSON, plain text, or self-contained HTML:

node tools/check.js fast --report=target/reports/checks.json --report=target/reports/checks.txt --report=target/reports/checks.html
node tools/scenario.js --report=target/reports/scenarios.json --report=target/reports/scenarios.html
node tools/scenario.js --profile ci --report=target/reports/perf.json --report=target/reports/perf.txt --report=target/reports/perf.html

All formats describe the same run, including partial results and failures. HTML opens offline and includes timings, metrics, correctness gates, and any budget comparison. check.js reports command-level status, timings, and commands left unexecuted after a failure; individual Cargo test output remains in the terminal. JSON retains the structured data used by automation and performance comparisons. Comparison inputs must be JSON; comparison output may use any of the three formats.

Parent directories are created automatically. Collaboration scenarios require new output filenames; checks, performance profiles, and comparisons atomically replace explicitly selected outputs. Baselines and comparison inputs cannot be overwritten. --dry-run writes no reports. Forced termination can prevent report publication.

Everyday development: fast checks

The default development gate is a focused smoke suite, not release qualification. Use Node.js 22+ (no npm installation) and the pinned Rust toolchain:

node tools/check.js fast

This runs:

The check runner's tests.
Source invariants (node tools/check.js invariants, described below).
Formatting and strict workspace Clippy with all targets/features.
A fresh debug build of the client/server binaries used by integration tests, with all features to match the surrounding development checks.
Workspace library/binary unit tests once, with all features.
The shared client contracts (local, TCP little-endian, and the big-endian test codec), CLI end-to-end tests, server acceptance/authorization tests, and the runner's end-to-end test (a fake web API handing jobs to scribe-runner against a real in-process storage server, including a job that publishes bundles in seven-byte parts through one injected 503).

No release build, benchmark compilation, performance workload, or duplicate full-workspace test configuration runs in this lane. A warm checkout should be much faster than full qualification; cold compilation still costs time. The runner prints elapsed time for every step and stops immediately on failure.

Run the new regression tests and relevant existing tests for the changed behavior as well. The smoke suite does not cover all integration tests. For complete coverage of affected packages in one invocation:

node tools/check.js fast --package scribe-client --package scribe-cli
node tools/check.js fast --package scribe-auth --package scribe-tls

--package (short form -p) is repeatable and is accepted only in fast mode. Selected packages run their complete all-feature test sets; their unit and smoke entries are excluded from earlier steps to avoid running them twice. Cargo validates package names. Package selection is explicit, so the runner does not pretend a filename heuristic can determine every affected caller. Choose downstream packages when a shared API or invariant changes.

For quick iteration before the final development gate, select an individual test:

cargo test --locked --all-features -p scribe-client --test contract get::
cargo test --locked --all-features -p scribe-cli --test cli_e2e get_downloads
cargo test --locked --all-features -p scribe-server-cli --test kill_publication

Use focused tests during the edit/debug loop, then run fast once when the change is ready for review. Do not repeat a passing gate without a new change, failure, or unresolved concern. Full qualification is separate; it is not the next step after a successful fast run.

When a run is unexpectedly slow, save a report and distinguish compilation from test execution. Cargo prints build time before each test binary; the runner reports each command's total time. A test that has finished its work can still hang in service teardown. The TCP and auth listeners explicitly set their cloned sockets nonblocking before passing them to Tokio: omitting that step caused Windows tests to hang in the accept loop instead of observing shutdown. Bounded idle-shutdown regressions cover both services.

The non-reading download regression also covers forced drain: Windows uses the cancellable copy transport because this implementation has no native file-to-socket transfer there. Switching it to a blocking copy held shutdown until the 20-second socket timeout even after the drain deadline expired.

CLI integration checks use debug executables. The client runs with an explicit 8 MiB stack budget because its generated parser overflowed the Windows default main-thread stack before even handling --version. The startup regression runs the executable itself; parser unit tests on Rust test threads did not expose the smaller executable stack. CLI integration diagnostics include the exit status of a child process that fails without output.

The platform detach regression (cargo test --locked --all-features -p scribe-platform --test detach) captures a short-lived parent's output while its detached descendant remains alive. This checks that Windows daemon startup does not retain the caller's output pipes through incidental handle inheritance. Controller lifecycle tests also exercise captured up --daemon commands.

Keep the same CARGO_TARGET_DIR and feature flags during iteration. The runner refreshes integration binaries with --all-features; switching that build to default features introduces another compilation configuration. Different package selections can still enable different dependency features. Concurrent Cargo commands sharing one target directory wait on its build lock rather than building in parallel. Incremental compilation is disabled by the repository configuration; CARGO_INCREMENTAL=1 is an optional local experiment for repeated edits to one crate, not a demonstrated improvement to the whole suite.

Changes to format bytes must run the affected golden fixtures. Storage, acceptance, auth, and platform changes need their relevant corruption, recovery, fault-injection, authorization, and platform tests. This is targeted coverage, not an instruction to run the entire release suite for every change. Documentation-only edits need link/example review, not Rust builds or tests. Runner/CI changes use node --test tools/check.test.js, dry-run review, and a fast run when the executed plan changes.

Source invariants

node tools/check.js invariants

This mode checks source rules that Clippy cannot express (AGENTS.md rule 17) over production sources under crates/*/src; tests are not scanned. It reports a violation when code outside crates/scribe-core/src/path.rs compares against the reserved .scribe or _state names directly instead of using scribe_core::path::is_reserved_scribe_name, is_reserved_state_name, or is_reserved_state_path; calls eq_ignore_ascii_case instead of folding through scribe_core::path::case_fold; or replaces a failed byte conversion (try_into, .ok(), or an aggregate fallback after try_from) with a default value instead of failing as Corrupt. Comments are ignored. Each finding names the file and line; the command exits 1 when any are found. Fast and full both run it as a step, and it can run alone.

Release or explicit on-demand qualification

node tools/check.js full

Full is the exhaustive gate: the check runner's tests, source invariants, formatting, strict Clippy, complete workspace tests with default features and with all features, benchmark compilation, fresh release CLI/server/perf binaries, and the CI performance correctness profile. When CARGO_TARGET_DIR is set, performance checks use that directory's freshly built release binaries. The default performance report remains under the repository's target/perf/ directory. The all-feature suite includes process-kill fault injection. Ignored large-scale tests are separate opt-in checks and do not run in either lane.

For timing regression checks, supply a baseline from comparable hardware instead of running the command above and then repeating the suite:

node tools/check.js full --baseline /path/to/js-performance-baseline.json

Without --baseline, full validates performance workload correctness but does not claim that timing budgets passed. Reports are written to target/perf/check-current.json (atomically replaced on each performance run). A private temporary run directory is removed after normal completion or handled interrupts. Forced termination can leave that directory behind; inspect it before explicitly cleaning it up.

Inspect either command plan without executing it:

node tools/check.js fast --dry-run
node tools/check.js full --dry-run
node tools/check.js --help

Do not automatically run full after fast. Require full for a release, or use it when explicitly requested for an investigation. Report whether fast, selected packages, or full actually ran, and which platforms were exercised. Certificate tests generate certificates themselves and require no external certificate tool.

What the tests check

Area	Coverage
File and network formats	Exact encoded bytes and content hashes, byte order, length limits, and rejection of malformed or unsupported data.
Object storage	Chunk reuse, loose and packed objects, corruption detection, interrupted writes, and rebuilding object locations.
Repository history	Accepted/rejected changes, repeat requests, locks, branches, merges, cross-branch lineage, divergent moves, filename reuse, snapshot pagination, index verification, and reopening/restoring stored history.
Client safety	Interrupted sync, edits made during transfer, file and directory collisions, symbolic links, and recovery of local work.
Client and server integration	Submit, sync, history, export, ignore rules, and branch workflows through both in-process and TCP connections.
Moves and rename merges	Multiple pending moves, dependencies and chained names; rename/edit, rename/delete, competing destinations, filename reuse, grouped choices, interruption, and abort recovery.
Lock policy and offline work	Accepted-main policy across branches, authorization and submit enforcement, offline intent, reconciliation, lock contention, and partial failures.
Historical downloads	Exact snapshot bytes, verified reconstruction, fresh destinations, and preservation of workspace state.
Unified status	One row per path, overlapping workflow facets, remote lock enrichment, offline/partial reports, inspection failures, and read-only behavior.
Concurrent connections	Responsiveness with idle clients, incomplete requests, stalled transfers, and waiting writes.
Authentication and TLS	Permissions, sessions, revocation, service accounts and their credential ceilings, automation and derived credentials, gateway delegation, namespace-wide reader grants, each service accepting only its own kind of client certificate, certificates minted with a copied CA key authenticating nothing, access-certificate renewal, fail-closed decoding of forged or damaged auth records, auth schema migration and its crash windows, offline administrator recovery and its crash window, rollback protection (the revision witness, the storage revision watch, and `scribe-auth restore`), keeping the last administrator credential, per-account credential limits, per-source pending-connection eviction, server trust, and withholding credentials until trust is verified.
Maintenance	Verification, corruption reporting, backup and restore, logical export and import, and explicit schema migrations.
Platform behavior	File I/O, data-root ownership locks, portable path rules, and platform-specific file access restrictions.
Cluster controller	`scribe-ctl setup` decisions fed a synthetic observation (no I/O), each variant set up, rerun, described by `where` and started with the real service binaries, a second cluster added to one file, and setup resumed after crashes inside its own write windows (`fault-injection`). `upgrade` decisions (set sanity, downgrades, data newer than the executables, plan and rollback order, resume) fed synthetic observations, and end to end between two real releases: the built executables and the same sources rebuilt with `SCRIBE_BUILD_ID=next` in `<target>/upgrade-next` (the test builds it with `cargo build --offline` on first use, incrementally afterwards), covering handover, backups, a store migration, an auth migration (with a wrapper `scribe-auth` reporting a newer schema) whose rollback restores the root with its revision witness, rollback after a failed health check and after a failed backup of a running cluster (the old set serves again), crash windows inside install, backup, switch and restore, the command lock, and refused sets. The docker runtime: Compose generation, path translation and docker plans as unit tests, and `tests/docker.rs` (Unix) drives setup, up, status, logs, upgrade and a rollback against a fake `docker` first on `PATH` that records every invocation, checking the exact command sequence; real Docker runs are manual. `renew` decisions (what is due, restart order, rollback range, resume and a stopped rollback) fed synthetic observations, and `tests/renew.rs` end to end: stand-in credentials one day from expiry replaced while a gateway cluster runs (only the holders restarting, auth serving throughout; and on a stopped one, started for the proof and stopped again), an injected verification failure that puts the old credential back and revokes the new, and a crash in every window inside renew, each resumed with exactly one credential issued. Remote operation: target parsing, quoting round trips through `sh -c`, command construction and push selection as unit tests, and `tests/remote.rs` (Unix) against a stand-in `ssh` first on `PATH` that runs the remote command locally in a temporary home with a clean environment (setup with bootstrap, the definition coming back, every lifecycle command, renew, exit codes, `ssh --`, a pushed release upgraded and handed over there, a pushed checkout's selection and the scribe-ctl a stand-in remote docker builds from it, rebuilt only when the checkout changes, one ssh connection per command through an emulated master, a remote setup held inside a step (`SCRIBE_FAULT=…=hold`) that finishes after its client is killed and its terminal hangs up, and a Ctrl-C that stops it at a safe point, resumably; `bootstrap` refreshing only the remote scribe-ctl beside a held setup, leaving `bin/current` and the cluster alone, and installing a release directory), with the remote controller's selection rule as a unit test; real ssh, its connection sharing and rsync 3 runs are manual. `doctor`: every classification fed synthetic facts (exit codes 0/1/3, rendering, the JSON document), and end to end a stopped and a running cluster (nothing written), a 0755 auth root, a pending journal, a credential near expiry, a remote report merged over one connection with nothing pushed, and after a fake-docker adopt the old project named with only read commands sent to docker. The ctl tests take their ports from blocks reserved under an OS file lock in the system temp directory (`tests/support::Ports`), so concurrent test processes never share a port.
Command output	Parseable JSON, logging configuration, and terminal output when redirected.

Fault injection requires the fault-injection Cargo feature and is armed by SCRIBE_FAULT in test builds. It is not a normal service configuration option. For the guarantees these tests exercise, see durability and recovery.

Run performance checks

The JS scenario harness owns performance profiles, services, datasets, reports, and regression comparison. The scribe-perf helper performs only direct Rust measurements: persistent-client mixed load, head latency probes, and object-store writes/reads with internal counters. Build the executables before a run:

cargo build --release --locked -p scribe-server-cli -p scribe-cli -p scribe-perf
node tools/scenario.js --profile ci --report ./target/perf/current.json
node tools/scenario.js --profile ci --dry-run

Profiles use an isolated, managed plain-TCP loopback server, preserving the performance harness's measurement scope. Authenticated TLS collaboration and compression matrices remain separate scenario plans under the same entry point. CLI measurements include process startup and connection setup; helper operations use persistent connections and time requests inside Rust. JS scheduling and JSON serialization are outside those request measurements.

The concurrent-workload watchdog includes working-copy preparation: its allowance scales with the client count and the slowest fresh sync observed earlier in the run, followed by the requested active duration and a bounded drain allowance. This setup allowance does not change the measured duration, correctness gates, or baseline timing budgets. Verbose output identifies preparation and active work separately.

The shared runtime creates a private directory beneath --temp-dir (the OS temp directory by default), owns every child, and removes run data after success, failure, or handled interrupts. Use --keep-data to retain it for diagnosis. Forced termination or power loss can prevent cleanup. Services and clients are isolated from your settings and credentials. Supply --bin-dir for executables outside target/release. No Cargo invocation occurs during measurement.

--report atomically replaces the specified performance report, including partial results after failures. Performance JSON reports use schema 2, record the dataset manifest hash, binary hashes, environment and run settings, and aggregate latency samples into p50/p95/p99 summaries. No report or baseline is updated implicitly except the explicitly selected output files. Repeat --report for multiple formats; budget comparison results are included in each output, even when timing budgets fail.

Compare two reports from comparable hardware with:

node tools/scenario.js --compare /path/to/js-performance-baseline.json --current ./target/perf/current.json
# Or compare immediately after one run:
node tools/scenario.js --profile ci --report ./target/perf/current.json --baseline /path/to/js-performance-baseline.json

The default JSON policy is tools/scenarios/performance/regression.json. Use --policy FILE for another policy. Comparison refuses mismatched datasets, transport, run settings, or environments, failed correctness gates, and missing budgeted metrics. Timing budgets need comparable hardware and an otherwise idle machine. The checked-in tools/scribe-perf/baselines/macos-ci.json uses report schema 1, which comparison refuses: capture and review a new baseline in an explicit commit before enabling timing comparisons.

ci includes imports, fresh and no-op sync, local edits and insertions, full rewrites, many small edits, historical export, bulk-transfer responsiveness, interrupted sync, mixed clients, metadata timings, restart, logging, backup/restore, and small-object storage. --repeat, --duration-secs, and --clients control sampling and load. million-paths and scale are opt-in workloads for dedicated hardware; --quick reduces million-paths to 100,000 paths. Scale requires an explicit --disk-budget-gib. Capacity checks include generated fixtures, archive, working copies, backup/restore, and concurrent readers; the estimate is not a disk quota. Large-file mutations use bounded buffers.

Focused harness checks (the integration checks need prebuilt binaries):

node --test tools/scenarios/performance/performance.test.js
node --test tools/scenarios/performance/integration.test.js
cargo test --locked --all-features -p scribe-perf

To test freshly built debug binaries, set SCRIBE_SCENARIO_BIN_DIR to the absolute target/debug path when invoking the Node integration tests. This is a test-only binary selector; the runner uses the explicit --bin-dir option.

Performance progress goes to stderr; stdout contains a JSON completion summary or comparison. A successful correctness run does not establish performance capacity. A run that misses the brief in-flight transfer window reports that limitation.

Measure synthetic concurrent capacity

The concurrency benchmark runs exact client-count ladders for head requests, tree pages, durable submit workflows, and mixed load. It saves throughput, tail latency, error/progress gates, and a capacity summary under explicit latency budgets. Every trial uses disposable data with cleanup on success, failure, and interruption.

cargo build --release --all-features --locked -p scribe-perf -p scribe-cli -p scribe-server-cli
node tools/concurrency.js --levels 1,4,16,64,128 --report target/perf/capacity.json --report target/perf/capacity.html
node --test tools/scenarios/timings.test.js tools/scenarios/concurrency/runner.test.js tools/scenarios/concurrency/integration.test.js

This is a local synthetic measurement, separate from release qualification and performance-baseline comparisons. See the guide for operation definitions, measurement windows, minimum samples, and cleanup guarantees. Add --timings for a diagnostic run using the shared profiling framework: the report correlates client/server phases and excludes setup traffic. Instrumented runs do not produce capacity claims; raw timing logs are cleaned with trial data.

Run collaboration scenarios

The scenario runner exercises authenticated collaboration and transfers with different storage and transfer compression settings. It requires Node.js 22 or newer, with no npm installation, plus the service and client executables:

cargo build --release --locked -p scribe-auth-cli -p scribe-server-cli -p scribe-cli
node tools/scenario.js --report ./target/scenario-results.json
node --test tools/scenario.test.js tools/scenarios/scenarios.test.js

Choose a new report filename for each run. The default scenarios create fresh repositories, users, and workspaces, then verify backup and restore before cleanup. The final command tests the runner itself. See the scenario runner documentation for configuration and additional workloads.

Check the password prompt on Unix

The interactive password check requires Python 3 and a Unix terminal environment:

cargo build --all-features --locked -p scribe-cli -p scribe-auth-cli
python3 crates/scribe-platform/tests/password_prompt.py

It checks hidden input, spaces and Unicode, saved sessions, cancellation, and restoration of terminal echo. On macOS it also checks Secure Input during entry and its restoration afterwards. This check does not cover Windows console input. The regular client tests cover redirected input and password files.

Platforms and limits

The repository's GitHub Actions workflow is started manually. Select fast (the default) for routine smoke checks or full for release/on-demand qualification. Both use the same runner as local development on Linux, Windows, and macOS. Dependency graphs are inspected in both modes. The emulated s390x big-endian job and performance report uploads run only in full mode. Hosted CI checks workload correctness; it does not compare timings against a developer machine's baseline. Before release, run full on this matrix and inspect all jobs.

Executables are built by the separate build workflow, also manual: choose all or a subset of the six executables and of the six platforms (linux-x64, linux-arm64, macos-arm64, macos-x64, windows-x64, windows-arm64, the names the web API uses for runner downloads). Linux executables are static musl builds through cross; the others build on the hosted runner of their operating system. Each platform uploads one archive per executable (<name>-<platform>.tar.gz or .zip) with a checksum file as a workflow artifact, and a tag input attaches them to that GitHub release, creating it as a draft when it does not exist. The selection rules live in tools/build-matrix.js and are tested by tools/build-matrix.test.js, which the fast suite runs.

The same executables build on a Scribe runner through the repository's own flows, .scribe/flows/build/{linux,macos,windows,all}.yml, which share the parts/build.yml job: scribe/checkout, rustup show, then node tools/build-dist.js with the run's binaries input (all, or a subset). The script builds the selected packages natively (a glibc Linux build, not the workflow's musl one), lays them out as dist/<name>-<platform>[.exe] with SHA256SUMS-<platform>.txt, and the flow publishes dist/** as bundles kept for ten runs. build/all has a job per platform label and so needs a runner of each kind; the single-platform flows need one. tools/build-dist.test.js covers the naming and selection.

Platform-specific tests run only on the matching operating system. A passing macOS run does not establish Linux or Windows coverage, and a configured CI job is not evidence that a particular revision has passed it.

Process termination tests do not prove durability under power loss or an operating-system crash. That requires VM or device-level interruption testing and verification of the recovered data. Performance reports count instrumented operations, not all kernel system calls. Authentication replication and failover are not implemented and have no test coverage.

Publication, trust and incremental-compaction regressions

The metadata tests pause archive publication while querying the prior published snapshot and verify that unarchived mutations have no acknowledgement. Staged batch tests inspect unchanged archive HEAD and independently restore the final publication. Snapshot tests include 50,000 files, long/deep paths, deletion and limit-plus-one boundaries. A persisted 3,000-level snapshot exercises live, changed, exact and point proofs across the real response byte budget. Format tests verify a single-entry proof larger than 8 MiB and a logical page visiting more than 8,192 nodes through bounded fragments, and refuse empty, replayed, altered and surplus fragments.

Client contracts cover tampered/omitted proof nodes, rollback refusal, explicit restore acceptance, backward sync with a newer history pin, and interrupted view changes. Fragmented sync contracts force node-boundary continuations through local and both TCP codecs, interrupt full and incremental planning, then verify retry and backward sync without moving the newer trust pin. Client unit tests reopen a workspace between bounded ancestry steps.

Server CLI tests kill real incremental-compaction processes at hard-link, locator, index, copy-cursor and store-swap boundaries, then resume through the command. Selective-rewrite tests retain accepted history and active upload pins under both zero and positive budgets. These fixtures establish correctness, not production multi-terabyte timing or release qualification on other platforms.