# Testing Scribe

Scribe's tests check that accepted history survives failures, local work is
protected during sync, and clients behave consistently across supported
platforms. This guide explains how to run the checks from a source checkout
and what their results can tell you.

## Save reports

Both JS entry points accept repeatable `--report=FILE` (or `--report FILE`).
The filename extension selects JSON, plain text, or self-contained HTML:

```sh
node tools/check.js fast --report=target/reports/checks.json --report=target/reports/checks.txt --report=target/reports/checks.html
node tools/scenario.js --report=target/reports/scenarios.json --report=target/reports/scenarios.html
node tools/scenario.js --profile ci --report=target/reports/perf.json --report=target/reports/perf.txt --report=target/reports/perf.html
```

All formats describe the same run, including partial results and failures. HTML
opens offline and includes timings, metrics, correctness gates, and any budget
comparison. `check.js` reports command-level status, timings, and commands left
unexecuted after a failure; individual Cargo test output remains in the terminal.
JSON retains the structured data used by automation and performance comparisons.
Comparison inputs must be JSON; comparison output may use any of the three formats.

Parent directories are created automatically. Collaboration scenarios require new
output filenames; checks, performance profiles, and comparisons atomically replace
explicitly selected outputs. Baselines and comparison inputs cannot be overwritten.
`--dry-run` writes no reports. Forced termination can prevent report publication.

## Everyday development: fast checks

The default development gate is a focused smoke suite, not release qualification.
Use Node.js 22+ (no npm installation) and the pinned Rust toolchain:

```sh
node tools/check.js fast
```

This runs:

1. The check runner's tests.
2. Source invariants (`node tools/check.js invariants`, described below).
3. Formatting and strict workspace Clippy with all targets/features.
4. A fresh debug build of the client/server binaries used by integration tests,
   with all features to match the surrounding development checks.
5. Workspace library/binary unit tests once, with all features.
6. The shared client contracts (local, TCP little-endian, and the big-endian test codec),
   CLI end-to-end tests, server acceptance/authorization tests, and the runner's
   end-to-end test (a fake web API handing jobs to `scribe-runner` against a real
   in-process storage server, including a job that publishes bundles in
   seven-byte parts through one injected `503`).

No release build, benchmark compilation, performance workload, or duplicate
full-workspace test configuration runs in this lane. A warm checkout should be
much faster than full qualification; cold compilation still costs time. The
runner prints elapsed time for every step and stops immediately on failure.

Run the new regression tests and relevant existing tests for the changed
behavior as well. The smoke suite does not cover all integration tests. For
complete coverage of affected packages in one invocation:

```sh
node tools/check.js fast --package scribe-client --package scribe-cli
node tools/check.js fast --package scribe-auth --package scribe-tls
```

`--package` (short form `-p`) is repeatable and is accepted only in fast mode.
Selected packages run their complete all-feature test
sets; their unit and smoke entries are excluded from earlier steps to avoid
running them twice. Cargo validates package names. Package selection is explicit,
so the runner does not pretend a filename heuristic can determine every affected
caller. Choose downstream packages when a shared API or invariant changes.

For quick iteration before the final development gate, select an individual test:

```sh
cargo test --locked --all-features -p scribe-client --test contract get::
cargo test --locked --all-features -p scribe-cli --test cli_e2e get_downloads
cargo test --locked --all-features -p scribe-server-cli --test kill_publication
```

Use focused tests during the edit/debug loop, then run fast once when the change
is ready for review. Do not repeat a passing gate without a new change, failure,
or unresolved concern. Full qualification is separate; it is not the next step
after a successful fast run.

When a run is unexpectedly slow, save a report and distinguish compilation from
test execution. Cargo prints build time before each test binary; the runner
reports each command's total time. A test that has finished its work can still
hang in service teardown. The TCP and auth listeners explicitly set their cloned
sockets nonblocking before passing them to Tokio: omitting that step caused
Windows tests to hang in the accept loop instead of observing shutdown. Bounded
idle-shutdown regressions cover both services.

The non-reading download regression also covers forced drain: Windows uses
the cancellable copy transport because this implementation has no native
file-to-socket transfer there. Switching it to a blocking copy held shutdown
until the 20-second socket timeout even after the drain deadline expired.

CLI integration checks use debug executables. The client runs with an explicit
8 MiB stack budget because its generated parser overflowed the Windows default
main-thread stack before even handling `--version`. The startup regression runs
the executable itself; parser unit tests on Rust test threads did not expose the
smaller executable stack. CLI integration diagnostics include the exit status
of a child process that fails without output.

The platform detach regression (`cargo test --locked --all-features -p
scribe-platform --test detach`) captures a short-lived parent's output while its
detached descendant remains alive. This checks that Windows daemon startup does
not retain the caller's output pipes through incidental handle inheritance.
Controller lifecycle tests also exercise captured `up --daemon` commands.

Keep the same `CARGO_TARGET_DIR` and feature flags during iteration. The runner
refreshes integration binaries with `--all-features`; switching that build to
default features introduces another compilation configuration. Different package
selections can still enable different dependency features. Concurrent Cargo
commands sharing one target directory wait on its build lock rather than building
in parallel. Incremental compilation is disabled by the repository configuration;
`CARGO_INCREMENTAL=1` is an optional local experiment for repeated edits to one
crate, not a demonstrated improvement to the whole suite.

Changes to format bytes must run the affected golden fixtures. Storage,
acceptance, auth, and platform changes need their relevant corruption, recovery,
fault-injection, authorization, and platform tests. This is targeted coverage,
not an instruction to run the entire release suite for every change.
Documentation-only edits need link/example review, not Rust builds or tests.
Runner/CI changes use `node --test tools/check.test.js`, dry-run review, and a
fast run when the executed plan changes.

### Source invariants

```sh
node tools/check.js invariants
```

This mode checks source rules that Clippy cannot express (AGENTS.md rule 17)
over production sources under `crates/*/src`; tests are not scanned. It
reports a violation when code outside `crates/scribe-core/src/path.rs`
compares against the reserved `.scribe` or `_state` names directly instead of
using `scribe_core::path::is_reserved_scribe_name`, `is_reserved_state_name`,
or `is_reserved_state_path`; calls `eq_ignore_ascii_case` instead of folding
through `scribe_core::path::case_fold`; or replaces a failed byte conversion
(`try_into`, `.ok()`, or an aggregate fallback after `try_from`) with a default
value instead of failing as `Corrupt`. Comments are ignored. Each finding names
the file and line; the command exits 1 when any are found. Fast and full both
run it as a step, and it can run alone.

## Release or explicit on-demand qualification

```sh
node tools/check.js full
```

Full is the exhaustive gate: the check runner's tests, source invariants,
formatting, strict Clippy, complete workspace tests with default features and
with all features, benchmark compilation, fresh release CLI/server/perf
binaries, and the CI performance correctness profile.
When `CARGO_TARGET_DIR` is set, performance checks use that directory's freshly
built release binaries. The default performance report remains under the
repository's `target/perf/` directory.
The all-feature suite includes process-kill fault injection. Ignored large-scale
tests are separate opt-in checks and do not run in either lane.

For timing regression checks, supply a baseline from comparable hardware instead
of running the command above and then repeating the suite:

```sh
node tools/check.js full --baseline /path/to/js-performance-baseline.json
```

Without `--baseline`, full validates performance workload correctness but does
not claim that timing budgets passed. Reports are written to
`target/perf/check-current.json` (atomically replaced on each performance run).
A private temporary run directory is removed after normal completion or handled
interrupts. Forced termination can leave that directory behind; inspect it before
explicitly cleaning it up.

Inspect either command plan without executing it:

```sh
node tools/check.js fast --dry-run
node tools/check.js full --dry-run
node tools/check.js --help
```

Do not automatically run full after fast. Require full for a release, or use it
when explicitly requested for an investigation. Report whether fast, selected
packages, or full actually ran, and which platforms were exercised. Certificate
tests generate certificates themselves and require no external certificate tool.

## What the tests check

| Area | Coverage |
| --- | --- |
| File and network formats | Exact encoded bytes and content hashes, byte order, length limits, and rejection of malformed or unsupported data. |
| Object storage | Chunk reuse, loose and packed objects, corruption detection, interrupted writes, and rebuilding object locations. |
| Repository history | Accepted/rejected changes, repeat requests, locks, branches, merges, cross-branch lineage, divergent moves, filename reuse, snapshot pagination, index verification, and reopening/restoring stored history. |
| Client safety | Interrupted sync, edits made during transfer, file and directory collisions, symbolic links, and recovery of local work. |
| Client and server integration | Submit, sync, history, export, ignore rules, and branch workflows through both in-process and TCP connections. |
| Moves and rename merges | Multiple pending moves, dependencies and chained names; rename/edit, rename/delete, competing destinations, filename reuse, grouped choices, interruption, and abort recovery. |
| Lock policy and offline work | Accepted-main policy across branches, authorization and submit enforcement, offline intent, reconciliation, lock contention, and partial failures. |
| Historical downloads | Exact snapshot bytes, verified reconstruction, fresh destinations, and preservation of workspace state. |
| Unified status | One row per path, overlapping workflow facets, remote lock enrichment, offline/partial reports, inspection failures, and read-only behavior. |
| Concurrent connections | Responsiveness with idle clients, incomplete requests, stalled transfers, and waiting writes. |
| Authentication and TLS | Permissions, sessions, revocation, service accounts and their credential ceilings, automation and derived credentials, gateway delegation, namespace-wide reader grants, each service accepting only its own kind of client certificate, certificates minted with a copied CA key authenticating nothing, access-certificate renewal, fail-closed decoding of forged or damaged auth records, auth schema migration and its crash windows, offline administrator recovery and its crash window, rollback protection (the revision witness, the storage revision watch, and `scribe-auth restore`), keeping the last administrator credential, per-account credential limits, per-source pending-connection eviction, server trust, and withholding credentials until trust is verified. |
| Maintenance | Verification, corruption reporting, backup and restore, logical export and import, and explicit schema migrations. |
| Platform behavior | File I/O, data-root ownership locks, portable path rules, and platform-specific file access restrictions. |
| Cluster controller | `scribe-ctl setup` decisions fed a synthetic observation (no I/O), each variant set up, rerun, described by `where` and started with the real service binaries, a second cluster added to one file, and setup resumed after crashes inside its own write windows (`fault-injection`). `upgrade` decisions (set sanity, downgrades, data newer than the executables, plan and rollback order, resume) fed synthetic observations, and end to end between two real releases: the built executables and the same sources rebuilt with `SCRIBE_BUILD_ID=next` in `<target>/upgrade-next` (the test builds it with `cargo build --offline` on first use, incrementally afterwards), covering handover, backups, a store migration, an auth migration (with a wrapper `scribe-auth` reporting a newer schema) whose rollback restores the root with its revision witness, rollback after a failed health check and after a failed backup of a running cluster (the old set serves again), crash windows inside install, backup, switch and restore, the command lock, and refused sets. The docker runtime: Compose generation, path translation and docker plans as unit tests, and `tests/docker.rs` (Unix) drives setup, up, status, logs, upgrade and a rollback against a fake `docker` first on `PATH` that records every invocation, checking the exact command sequence; real Docker runs are manual. `renew` decisions (what is due, restart order, rollback range, resume and a stopped rollback) fed synthetic observations, and `tests/renew.rs` end to end: stand-in credentials one day from expiry replaced while a gateway cluster runs (only the holders restarting, auth serving throughout; and on a stopped one, started for the proof and stopped again), an injected verification failure that puts the old credential back and revokes the new, and a crash in every window inside renew, each resumed with exactly one credential issued. Remote operation: target parsing, quoting round trips through `sh -c`, command construction and push selection as unit tests, and `tests/remote.rs` (Unix) against a stand-in `ssh` first on `PATH` that runs the remote command locally in a temporary home with a clean environment (setup with bootstrap, the definition coming back, every lifecycle command, renew, exit codes, `ssh --`, a pushed release upgraded and handed over there, a pushed checkout's selection and the scribe-ctl a stand-in remote docker builds from it, rebuilt only when the checkout changes, one ssh connection per command through an emulated master, a remote setup held inside a step (`SCRIBE_FAULT=…=hold`) that finishes after its client is killed and its terminal hangs up, and a Ctrl-C that stops it at a safe point, resumably; `bootstrap` refreshing only the remote scribe-ctl beside a held setup, leaving `bin/current` and the cluster alone, and installing a release directory), with the remote controller's selection rule as a unit test; real ssh, its connection sharing and rsync 3 runs are manual. `doctor`: every classification fed synthetic facts (exit codes 0/1/3, rendering, the JSON document), and end to end a stopped and a running cluster (nothing written), a 0755 auth root, a pending journal, a credential near expiry, a remote report merged over one connection with nothing pushed, and after a fake-docker adopt the old project named with only read commands sent to docker. The ctl tests take their ports from blocks reserved under an OS file lock in the system temp directory (`tests/support::Ports`), so concurrent test processes never share a port. |
| Command output | Parseable JSON, logging configuration, and terminal output when redirected. |

Fault injection requires the `fault-injection` Cargo feature and is armed by
`SCRIBE_FAULT` in test builds. It is not a normal service configuration option.
For the guarantees these tests exercise, see
[durability and recovery](/docs/durability-and-recovery.md).

## Run performance checks

The JS scenario harness owns performance profiles, services, datasets, reports,
and regression comparison. The `scribe-perf` helper performs only direct Rust
measurements: persistent-client mixed load, head latency probes, and object-store
writes/reads with internal counters. Build the executables before a run:

```sh
cargo build --release --locked -p scribe-server-cli -p scribe-cli -p scribe-perf
node tools/scenario.js --profile ci --report ./target/perf/current.json
node tools/scenario.js --profile ci --dry-run
```

Profiles use an isolated, managed plain-TCP loopback server, preserving the
performance harness's measurement scope. Authenticated TLS collaboration and
compression matrices remain separate scenario plans under the same entry point.
CLI measurements include process startup and connection setup; helper operations
use persistent connections and time requests inside Rust. JS scheduling and JSON
serialization are outside those request measurements.

The concurrent-workload watchdog includes working-copy preparation: its allowance
scales with the client count and the slowest fresh sync observed earlier in the
run, followed by the requested active duration and a bounded drain allowance.
This setup allowance does not change the measured duration, correctness gates,
or baseline timing budgets. Verbose output identifies preparation and active
work separately.

The shared runtime creates a private directory beneath `--temp-dir` (the OS temp
directory by default), owns every child, and removes run data after success,
failure, or handled interrupts. Use `--keep-data` to retain it for diagnosis.
Forced termination or power loss can prevent cleanup. Services and clients are
isolated from your settings and credentials. Supply `--bin-dir` for executables
outside `target/release`. No Cargo invocation occurs during measurement.

`--report` atomically replaces the specified performance report, including partial
results after failures. Performance JSON reports use schema 2, record the dataset manifest hash,
binary hashes, environment and run settings, and aggregate latency samples into
p50/p95/p99 summaries. No report or baseline is updated implicitly except the
explicitly selected output files. Repeat `--report` for multiple formats; budget
comparison results are included in each output, even when timing budgets fail.

Compare two reports from comparable hardware with:

```sh
node tools/scenario.js --compare /path/to/js-performance-baseline.json --current ./target/perf/current.json
# Or compare immediately after one run:
node tools/scenario.js --profile ci --report ./target/perf/current.json --baseline /path/to/js-performance-baseline.json
```

The default JSON policy is
[`tools/scenarios/performance/regression.json`](https://github.com/Flying-Rat/scribe/blob/main/tools/scenarios/performance/regression.json).
Use `--policy FILE` for another policy. Comparison refuses mismatched datasets,
transport, run settings, or environments, failed correctness gates, and missing
budgeted metrics. Timing budgets need comparable hardware and an otherwise idle
machine. The checked-in `tools/scribe-perf/baselines/macos-ci.json` uses report
schema 1, which comparison refuses: capture and review a new baseline in an
explicit commit before enabling timing comparisons.

`ci` includes imports, fresh and no-op sync, local edits and insertions, full
rewrites, many small edits, historical export, bulk-transfer responsiveness,
interrupted sync, mixed clients, metadata timings, restart, logging, backup/restore,
and small-object storage. `--repeat`, `--duration-secs`, and `--clients` control
sampling and load. `million-paths` and `scale` are opt-in workloads for dedicated
hardware; `--quick` reduces million-paths to 100,000 paths. Scale requires an
explicit `--disk-budget-gib`. Capacity checks include generated fixtures, archive,
working copies, backup/restore, and concurrent readers; the estimate is not a disk
quota. Large-file mutations use bounded buffers.

Focused harness checks (the integration checks need prebuilt binaries):

```sh
node --test tools/scenarios/performance/performance.test.js
node --test tools/scenarios/performance/integration.test.js
cargo test --locked --all-features -p scribe-perf
```

To test freshly built debug binaries, set `SCRIBE_SCENARIO_BIN_DIR` to the
absolute `target/debug` path when invoking the Node integration tests. This is a
test-only binary selector; the runner uses the explicit `--bin-dir` option.

Performance progress goes to stderr; stdout contains a JSON completion summary or
comparison. A successful correctness run does not establish performance capacity.
A run that misses the brief in-flight transfer window reports that limitation.

## Measure synthetic concurrent capacity

The [concurrency benchmark](/docs/concurrency-benchmark.md) runs exact client-count
ladders for head requests, tree pages, durable submit workflows, and mixed load.
It saves throughput, tail latency, error/progress gates, and a capacity summary
under explicit latency budgets. Every trial uses disposable data with cleanup
on success, failure, and interruption.

```sh
cargo build --release --all-features --locked -p scribe-perf -p scribe-cli -p scribe-server-cli
node tools/concurrency.js --levels 1,4,16,64,128 --report target/perf/capacity.json --report target/perf/capacity.html
node --test tools/scenarios/timings.test.js tools/scenarios/concurrency/runner.test.js tools/scenarios/concurrency/integration.test.js
```

This is a local synthetic measurement, separate from release qualification and
performance-baseline comparisons. See the guide for operation definitions,
measurement windows, minimum samples, and cleanup guarantees.
Add `--timings` for a diagnostic run using the shared profiling framework: the
report correlates client/server phases and excludes setup traffic. Instrumented
runs do not produce capacity claims; raw timing logs are cleaned with trial data.

## Run collaboration scenarios

The scenario runner exercises authenticated collaboration and transfers with
different storage and transfer compression settings. It requires Node.js 22 or
newer, with no npm installation, plus the service and client executables:

```sh
cargo build --release --locked -p scribe-auth-cli -p scribe-server-cli -p scribe-cli
node tools/scenario.js --report ./target/scenario-results.json
node --test tools/scenario.test.js tools/scenarios/scenarios.test.js
```

Choose a new report filename for each run. The default scenarios create fresh
repositories, users, and workspaces, then verify backup and restore before
cleanup. The final command tests the runner itself. See the
[scenario runner documentation](https://github.com/Flying-Rat/scribe/blob/main/tools/scenarios/README.md) for configuration
and additional workloads.

## Check the password prompt on Unix

The interactive password check requires Python 3 and a Unix terminal environment:

```sh
cargo build --all-features --locked -p scribe-cli -p scribe-auth-cli
python3 crates/scribe-platform/tests/password_prompt.py
```

It checks hidden input, spaces and Unicode, saved sessions, cancellation, and
restoration of terminal echo. On macOS it also checks Secure Input during entry
and its restoration afterwards. This check does not cover Windows console input.
The regular client tests cover redirected input and password files.

## Platforms and limits

The repository's GitHub Actions workflow is started manually. Select `fast`
(the default) for routine smoke checks or `full` for release/on-demand
qualification. Both use the same runner as local development on Linux, Windows,
and macOS. Dependency graphs are inspected in both modes. The emulated s390x
big-endian job and performance report uploads run only in full mode. Hosted CI
checks workload correctness; it does not compare timings against a developer
machine's baseline. Before release, run full on this matrix and inspect all jobs.

Executables are built by the separate `build` workflow, also manual: choose
`all` or a subset of the six executables and of the six platforms (`linux-x64`,
`linux-arm64`, `macos-arm64`, `macos-x64`, `windows-x64`, `windows-arm64`, the
names the web API uses for runner downloads). Linux executables are static
musl builds through `cross`; the others build on the hosted runner of their
operating system. Each platform uploads one archive per executable
(`<name>-<platform>.tar.gz` or `.zip`) with a checksum file as a workflow
artifact, and a `tag` input attaches them to that GitHub release, creating it
as a draft when it does not exist. The selection rules live in
`tools/build-matrix.js` and are tested by `tools/build-matrix.test.js`, which
the fast suite runs.

The same executables build on a Scribe runner through the repository's own
flows, `.scribe/flows/build/{linux,macos,windows,all}.yml`, which share the
`parts/build.yml` job: `scribe/checkout`, `rustup show`, then
`node tools/build-dist.js` with the run's `binaries` input (all, or a subset).
The script builds the selected packages natively (a glibc Linux build, not the
workflow's musl one), lays them out as `dist/<name>-<platform>[.exe]` with
`SHA256SUMS-<platform>.txt`, and the flow publishes `dist/**` as bundles kept
for ten runs. `build/all` has a job per platform label and so needs a runner
of each kind; the single-platform flows need one. `tools/build-dist.test.js`
covers the naming and selection.

Platform-specific tests run only on the matching operating system. A passing
macOS run does not establish Linux or Windows coverage, and a configured CI job
is not evidence that a particular revision has passed it.

Process termination tests do not prove durability under power loss or an
operating-system crash. That requires VM or device-level interruption testing
and verification of the recovered data. Performance reports count instrumented
operations, not all kernel system calls. Authentication replication and failover
are not implemented and have no test coverage.

### Publication, trust and incremental-compaction regressions

The metadata tests pause archive publication while querying the prior published
snapshot and verify that unarchived mutations have no acknowledgement. Staged
batch tests inspect unchanged archive HEAD and independently restore the final
publication. Snapshot tests include 50,000 files, long/deep paths, deletion and
limit-plus-one boundaries. A persisted 3,000-level snapshot exercises live,
changed, exact and point proofs across the real response byte budget. Format tests verify a single-entry
proof larger than 8 MiB and a logical page visiting more than 8,192 nodes through
bounded fragments, and refuse empty, replayed, altered and surplus fragments.

Client contracts cover tampered/omitted proof nodes, rollback refusal, explicit
restore acceptance, backward sync with a newer history pin, and interrupted view
changes. Fragmented sync contracts force node-boundary continuations through
local and both TCP codecs, interrupt full and incremental planning, then verify
retry and backward sync without moving the newer trust pin. Client unit tests
reopen a workspace between bounded ancestry steps.

Server CLI tests kill real incremental-compaction processes at hard-link, locator,
index, copy-cursor and store-swap boundaries, then resume through the command.
Selective-rewrite tests retain accepted history and active upload pins under both
zero and positive budgets. These fixtures establish correctness, not production
multi-terabyte timing or release qualification on other platforms.